CTest able to daemonize fixtures?

Thanks for the work on CMake, it’s a huge help with cross-platform build.

We have been using CMake for a long time and at the moment I am trying to transfer our bash scripted testing into CTest… it can handle simple “one-sided” tests such as run a test http client app that performs test transaction against a remote server very well.

But the same set of tests need checking again against an instance of the built http server running locally instead of a remote server: these are “two sided” tests then, requiring orchestration of the local peer daemon as well as the apps under test. We spawn the peer that the actual test will test against before running those tests and take it down afterwards.

FIXTURES_REQUIRED seems to have the right idea but as far as I can tell (maybe I just use it wrong), it’s limited to running a single executable in blocking mode like the normal test flow, and if I put the daemon spawn as an add_test() the normal way it wants to actually run that as a test itself.

Is there a correct way to express this with CTest? Since we work on windows as well as *nix and other platforms, it’s highly desirable if we can express this relationship in the test flow at CTest level.

There is no direct support for running a daemonized test through CTest, but it can be done. It takes two parts:

  • Have a test case that is a setup step for your test fixture (FIXTURES_SETUP). Implement it as a wrapper script that your test case runs runs. The script needs to run the thing you want to daemonize in the background, but importantly it also needs to record somewhere the process ID of that daemonized process. Writing it to a file at a known name and location is usually what you want to do for this, but I recommend that be a name that is at least specific to the fixture name or other sufficiently unique name.
  • Have another test that is a cleanup step for your test fixture (FIXTURES_CLEANUP). This will be another script, only this time it’s purpose is to read in the file created in the setup test to provide the process ID and then kill that process.

I’ve used the above strategy at a previous company with great success, although in that case we were able to assume Unix only, which made the wrapper scripts easier to write. Doing it in a cross-platform way will be more work. We were able to generalise this enough to be able to re-use much of the work to launch and stop various servers as part of fixtures (i.e. we wrote custom CMake commands which took a few arguments and wrote the scripts for us).

Thanks for the reply. I tried to implement what I understood you’re suggesting just for Linux initially… I created a shim shellscript

$ cat ../scripts/ctest-background.sh
#!/bin/bash

$1 $2 $3 $3 $5 $6 $7 $8 $9 &
exit 0

and use the following scheme

add_test(NAME slocalserver COMMAND ../../../scripts/ctest-background.sh $<TARGET_FILE:lws-minimal-http-server-tls>)
        add_test(NAME clocalserver COMMAND killall lws-minimal-http-server-tls)

        set_property(TEST slocalserver PROPERTY WORKING_DIRECTORY ../minimal-examples/http-server/minimal-http-server-tls)
        set_property(TEST slocalserver PROPERTY FIXTURES_SETUP localserver)
        set_property(TEST clocalserver PROPERTY FIXTURES_CLEANUP localserver)

        add_test(NAME http-client-multi COMMAND lws-minimal-http-client-multi -l)
        set_tests_properties(http-client-multi PROPERTIES FIXTURES_REQUIRED localserver)

But ctest stops at the point of creating the server, as if it was still waiting for its spawned process exit

      Start 14: slocalserver

I can see that the spawned process exists OK.

2598941 pts/0    S+     0:00  \_ /projects/libwebsockets/bb/bin/lws-minimal-http-server-tls

Did I miss the point or mangle the implementation?

That seems to be in the ballpark of what I was trying to describe. ctest could still be blocking potentially because the output pipes are still open, but that’s a guess. I don’t have access to the implementation any more, so I can’t check what we did on the spawning side. You could try redirecting stderr and stdout within the script to see if that prevents ctest from still blocking.

Before I forget, beware of using killall on the cleanup side. I know this is just trying to get things working for now, but I recall also starting that way and realising after a while that you really do need to go by the PID. You may have multiple instances of the executable running, you could have multiple people running jobs on the same machine, etc.

Thanks Craig, it was a good guess. It acts well with the shim script using this

$1 $2 $3 $4 $5 $6 $7 $8 $9 2>/dev/null 1>/dev/null 0</dev/null &
sleep 1

… where the sleep is a not very beautiful way to ensure it had a chance to start before we try to talk to it.

I take the point about killall. But with targeting the specific pid it can also meet a situation where the test action was interrupted or killed and it wasn’t cleaned up, leading to immortal processes blocking further tests because it’s listening on the port. Maybe CMake always runs the cleanup step even on SIGINT it does seem to trap it somewhat.

I’ll try to extend this to windows later… it’d be a cool feature if there was a BACKGROUND property or somesuch available for tests and no need to get involved in the platform differences.

Yes we had that exact problem to overcome too. :slight_smile: The startup script should be able to look for a pre-existing PID file and check if that process is still running. If it isn’t, then ignore that PID and start a new process. A dead process shouldn’t be holding onto the port, at least not for all that long, and if it did there’s nothing we can do about it from the ctest side anyway. If the previous process corresponding to the PID is still running, the script should fail the test case, which will prevent all other tests requiring that fixture to be skipped.

In case it is useful, when working on a particular test that needed a server running, I’d use the -FS, -FC and -FA options to ctest to avoid running the setup and/or cleanup parts of the fixture. I would use -FC to prevent the cleanup test to deliberately leave the server running so that I could run my specific test case under a debugger, or re-run it repeatedly without having to go through the setup/cleanup each time. I’d use -FA where I wanted to re-run it through ctest. And when I was done, I’d use -FS to allow ctest to finally clean up the running server.

It would indeed be good to have support for this in CMake itself. Rather than adding a BACKGROUND property to a test though, I’m thinking this might be better implemented as a CMake module. It would be better to keep add_test() focused on creating a single test case like it does now. The functionality required here needs to create two tests, one to start and the other to cleanup. That could be done in a command provided by some new module and it could be implemented with existing functionality. If you’re interested in having a go at this, contact me privately and I will see if I can put you in touch with past colleagues who might be able to give you some further guidance (you can use the contact details from here)

Maybe I misunderstand but add_test() is already officially overloaded in order to specify the fixture-related processes… setting up and tearing down the fixture is not a “test” but you must specify it like one. I realize these kinds of choice can be a bit fine and are often best made by a maintainer giving consideration to things an individual user might not care enough about, but putting a property on it doesn’t make anything worse than it already is wrt violating what add_test() is supposed to do, and it’s consistent with what is already there.

In my case (I think it’s representative of any client / server network type case) the background app is a server that is built from the same commit that’s under test, it also is as prone to failure as the clientside pieces built from the same sources and ideally needs to be watched as carefully with its output captured persistently same as the code that’s directly under test (and optionally dumped if that side fails same as the foreground test).

For windows case, I can’t find a way to background-spawn the server side that actually works trying to do it from the current add_test() either, I tried variations around cmd.exe /c start /b <target path>.

If you’re interested in having a go at this

I’m not against the idea if the result actually scratches my itch.

What I meant was that add_test() always creates a single test case and I think it should stay that way. The functionality we are talking about here requires adding at least two test cases, one to launch the server as a setup for the fixture and another test case to kill it as a cleanup for the fixture. A function provided by a module could take care of creating the two test cases so that project code only had to call one command to have both test cases created. That’s what I was trying to suggest earlier.

I also now recall that we usually had a third test case to check the output of the server, scanning for keywords like exception, warning, error, etc. The launching test case had to pipe the output of the server to a log file, since ctest itself can’t capture it because the test case ends once the server has been put in the background. We added a separate test case for checking and printing the log file if it detected any of the keywords so that we could see the output when something went wrong, but you could just get the cleanup test case to dump the output rather than having a third test case. I don’t recall why we did it as a separate test case, it might have been something to do with trying to reuse the FAIL_REGULAR_EXPRESSION test property, but I’m not sure. Scanning for keywords in the output would not always be appropriate in general though, some servers might legitimately log those keywords and it not be an error, so we’d probably be best to let projects set those fail regular expressions if they want such behavior.

I don’t know either off the top of my head. I’d focus more on doing it from within a script though rather than trying to do it directly as a ctest COMMAND. It will also be easier to work on as a script since you can invoke it directly without having to go through the whole ctest workflow.

It will also be easier to work on as a script

That’s a reasonable approach if it’s just unix-land. But “a script” will be at least two separate scripts if you are also having to support windows-land. And I’m not sure why a script will make it work when spawning cmd with args doesn’t, although I’m not sure why that doesn’t work atm either, so who knows. CTest already provides a nice platform-independent way to manage spawning things as a built-in primitive, it feels reasonable to extend that and continue to conceal the platform details in the same place.

The functionality we are talking about here requires adding at least two test cases, one to launch the server as a setup for the fixture and another test case to kill it as a cleanup for the fixture.

Right… CMake already understands there are “fixture” actions external to the foreground test flow that must be orchestrated as properties of the tests; it understands these actions persist for the duration of tests that depend on them since it has the concept of undoing them after the tests that depend on the fixture complete. The official, built-in way to code it today without backgrounding looks like what you describe already. Backgrounding stuff as a property / crossplatform primitive seems to be completely consonant with CTest spawning things crossplatform already and having the “persistent setup and teardown” concept expressed inside an overload of add_test() meaning as today.

A function provided by a module could take care of creating the two test cases so that project code only had to call one command to have both test cases created.

Well, OK, but it’s basically a macro on top of what would have to be able to be done by hand elaborated out. It doesn’t help guide us how it should be done underneath.

check the output of the server

Right… I think that approach that both the background fixture and the foreground code are under test is really the correct way to look at it… often they’re going to be built from the same codebase that’s under test and what we want to test is different bits of the codebase interacting at runtime. Then it seems to lead to the idea that the crossplatform spawning / monitoring / reaping / io capture services already provided by CTest ought to have some parity between provision for foreground test and background fixture, since they are quite profoundly all part of a multi-process test action.

It’s certainly going to be easy to explain to someone familiar with the existing fixture arrangements that there’s just another property and everything else the same to spawn one or more fixture daemons for the test to interact with, and CTest takes care of the crossplatform implications and lifecycle. There is no need for any modules or macros or whatever in that case.

It sounds like we both meet the same situation testwise and feel CTest should handle it, but just differ on whether background fixtures are enough of a primitive that it should actually go in CTest.

I’ve mostly just been describing the approach I’ve used in the past and how it could be extended to cover all platforms. It could be done with existing CMake versions if you wanted to implement it as your own module. I’m not locked in to that approach, it’s just the least risky path and can be done by anyone without having to modify CMake’s internals.

I’m certainly willing to consider alternatives and I am open to the sort of arrangement you are describing. Personally, I don’t have the bandwidth to do it myself, but if others want to give it a go, I can give some thoughts here. What follows is just off the top of my head, I haven’t thought too deeply about the implications. That’s what discussion forums are for though, right? :wink:

One possibility is that we add a new boolean test property which says whether the test executable should be launched as a background process or not. It would be false by default to preserve backward compatibility. If it was set to true, then ctest would launch the executable and pipe its output to somewhere with a unique name, and once the process started successfully, the test case would be considered “done” (more on that below). I’ll refer to that test case as “Pre” in comments that follow. ctest would remember the process ID of the executable it launched so it could track its lifetime. I will assume for now that if using this, then FIXTURES_SETUP cannot be empty for “Pre” (I explain why further below). There’s a few choices for what happens after that:

  • If the executable for “Pre” crashes at some point, then any subsequent test cases that require that fixture will not be started. They will be deemed to have failed fixture dependencies. Any already-completed test cases that required the fixture would be unaffected. Not sure about any test cases already in progress. That would need further discussion.

  • Once all test cases that require the fixture are finished, ctest could automatically stop the background process. How it does that is uncertain though. Perhaps it should send it a SIGTERM to ask it to end, then if that takes too long, force-kill it (not sure if that qualifies as pass or fail for “Pre”).

  • Once the executable launched for “Pre” has ended, only then would the results for the “Pre” test case be recorded. It should support the usual PASS_REGULAR_EXPRESSION, FAIL_REGULAR_EXPRESSION logic. Some test properties might not make sense (e.g. SKIP_REGULAR_EXPRESSION and SKIP_RETURN_CODE could no longer work). All redirected output recorded for “Pre” could/should be logged as one at this point.

  • There would need to be some way to leave the executable running even after ctest ends. This is a feature I personally used a lot with the way I had this supported with scripts. It is very useful during development while working on a test case that needs the fixture. I think perhaps we should force such tests to also be cleanups for the fixtures that they are setups for. That would allow the developer to turn off the “kill the process” part using the existing ctest -FC command line option, which would satisfy this dot point’s requirement.

  • We have to think about the scenario where ctest is killed part-way through running a set of tests, with the executable from “Pre” still left running. The next time ctest is run, it will try to start up another executable again, but it may be blocked by the existing one. One a dev machine, this is easy to deal with, but on CI servers it is very problematic. I recall having significant issues with this in the past. We would sometimes find many orphan processes still running and taking up memory, we would sometimes see servers fail to start up and it was hard to see why they were failing. One can’t easily tell heuristically if there is a left-over process still running and you can’t just go randomly killing processes that look like yours by name (e.g. other people could be on the same machine and running their own builds of the same project, but in different directories to yours).

  • Not sure what impact this would have on the logic for the DEPENDS test property. I guess similar treatment to fixtures in that once the process has been launched, anything that DEPENDed on it should be allowed to start.

This post is already super long, but hopefully it is useful if someone wants to explore further. I suggest raising a ticket in gitlab to get agreement on an approach before going too far though.

https://gitlab.kitware.com/cmake/cmake/-/issues