tests that are meant to abort()

#1

I understand that the WILL_FAIL test property informs CTest that a test is expected to fail, which in this case means “exits with a non-zero exit code”. However, we have a set of tests that confirm that our application outright aborts when it is supposed to.

How can I inform CTest that this is the desired outcome?

#2

How can something outside of your test case confirm that your application aborted? Does it give a specific exit code or does it print something to stdout/stderr? You have to identify something that an outside observer can check. CTest can look for regular expression matches in the output, if there’s something that is always printed on abort. Otherwise, you’re left with checking the exit code. If neither of those are possible, you may need to explore deeper. For example, the GoogleTest framework supports death tests, so using a testing framework with similar features might give you what you’re looking for.

#3

At least on Linux, the status code is 134 and it prints “Aborted (core dumped)”. CTest clearly has some way of recognizing this: that SIGABRT was raised during the application, causing an abnormal exit. It seems to treat this as a third outcome in addition to pass or fail and so annotating with WILL_FAIL does nothing.

To be clear, when I say “aborted”, I mean that the application called std::abort(). I’m not doing anything to intercept the resulting SIGABRT, but by default it exits with a non-zero status code on all the platforms I’m familiar with. I’m not sure what the standard says, if anything.

#4

Linking in an object library consisting of a single source file with:

#include <cstdlib>
#include <csignal>

// This is a hack to implement death tests in CTest.
extern "C" void error_test_handle_abort(int) {
    std::_Exit(EXIT_FAILURE);
}

struct test_override_abort {
    test_override_abort() noexcept {
        std::signal(SIGABRT, error_test_handle_abort);
    }
};

test_override_abort handler{};

seems to hack around it. I imagine this is something similar to what Google Test does, but with an ugly unused global variable. I think there should be first-class support for death tests in CTest, though. WILL_FAIL should be supplemented by a new WILL_ABORT property.

#5

@craig.scott - I wonder if you know of any alternatives to the above workaround?

#6

It’s a bit odd to explicitly expect an application to abort. There could be any number of reasons why it might abort/crash and it’s not normal for that to be expected behavior! That said, one possibility is maybe you could wrap it in a call like ctest --build-and-test somehow. I haven’t thought it through and maybe it’s a completely wrong-headed, over-engineered way to tackle it, but it’s something you could explore at least. I’m a bit short on time right now, so I’ll have to leave you to investigate further.

1 Like
#7

No worries and thanks very much for getting back to me! Your book has been invaluable in retrofitting cmake to this project, by the way :slight_smile:

1 Like