tests that are meant to abort()

Code
1

I understand that the WILL_FAIL test property informs CTest that a test is expected to fail, which in this case means “exits with a non-zero exit code”. However, we have a set of tests that confirm that our application outright aborts when it is supposed to.

How can I inform CTest that this is the desired outcome?

2

How can something outside of your test case confirm that your application aborted? Does it give a specific exit code or does it print something to stdout/stderr? You have to identify something that an outside observer can check. CTest can look for regular expression matches in the output, if there’s something that is always printed on abort. Otherwise, you’re left with checking the exit code. If neither of those are possible, you may need to explore deeper. For example, the GoogleTest framework supports death tests, so using a testing framework with similar features might give you what you’re looking for.

3

At least on Linux, the status code is 134 and it prints “Aborted (core dumped)”. CTest clearly has some way of recognizing this: that SIGABRT was raised during the application, causing an abnormal exit. It seems to treat this as a third outcome in addition to pass or fail and so annotating with WILL_FAIL does nothing.

To be clear, when I say “aborted”, I mean that the application called std::abort(). I’m not doing anything to intercept the resulting SIGABRT, but by default it exits with a non-zero status code on all the platforms I’m familiar with. I’m not sure what the standard says, if anything.

4

Linking in an object library consisting of a single source file with:

#include <cstdlib>
#include <csignal>

// This is a hack to implement death tests in CTest.
extern "C" void error_test_handle_abort(int) {
    std::_Exit(EXIT_FAILURE);
}

struct test_override_abort {
    test_override_abort() noexcept {
        std::signal(SIGABRT, error_test_handle_abort);
    }
};

test_override_abort handler{};

seems to hack around it. I imagine this is something similar to what Google Test does, but with an ugly unused global variable. I think there should be first-class support for death tests in CTest, though. WILL_FAIL should be supplemented by a new WILL_ABORT property.

5

@craig.scott - I wonder if you know of any alternatives to the above workaround?

6

It’s a bit odd to explicitly expect an application to abort. There could be any number of reasons why it might abort/crash and it’s not normal for that to be expected behavior! That said, one possibility is maybe you could wrap it in a call like ctest --build-and-test somehow. I haven’t thought it through and maybe it’s a completely wrong-headed, over-engineered way to tackle it, but it’s something you could explore at least. I’m a bit short on time right now, so I’ll have to leave you to investigate further.

1 Like
7

No worries and thanks very much for getting back to me! Your book has been invaluable in retrofitting cmake to this project, by the way :slight_smile:

1 Like
8

There could be any number of reasons why it might abort/crash and it’s not normal for that to be expected behavior!

It’s totally reasonable to have tests that are “expected to fail”, e.g., we expect this code to have an assert() fail if we pass in this set of malformed inputs. Since assert() often uses SIGABRT, being able to catch these is highly desirable. (I’m pretty sure that GTest allows for catching expected-SIGABRT via the EXPECT_DEATH/ASSERT_DEATH macros.)

9

(I’m pretty sure that GTest allows for catching expected-SIGABRT via the EXPECT_DEATH/ASSERT_DEATH macros.)

The _DEATH macros are wrappers for the _EXIT macros, which use platform-specific APIs (fork on NIX and CreateProcess on Windows) to run the statement in a child process, isolating the parent process from its errors.

CTest never sees the SIGABRT when using GTest.

See: https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#how-it-works

10

I have an analogous issue with WILL_FAIL and segfaults. I made an interface to a large external library out of my control, and I want to test that bad inputs do fail. Sometimes (shaky) the external library segfaults, which then causes CTest to fail despite WILL_FAIL. Most of the time the external library doesn’t segfault, but returns an error code, which of course works OK with WILL_FAIL
My workaround was to skip these cases on CI and remember to run them locally.

However, I can see that others might expect that segfaults are indeed a “third outcome” that isn’t desirable.

11

Discovering/hijacking this a few years later (…), I can add that processes raising SIGABRT (e.g. using asserts which call abort()) will cause ctest to print Exception:Subprocess aborted and not show the output from the failing program despite how hard I try to make it: passing --extra-verbose or setting CTEST_OUTPUT_ON_FAILURE=1 in the environment won’t make a single bit of a difference.

Then when I switched the assertion code in my project to use exit(1) instead, it instantly started working.

Is this by design or is this a limitation in ctest/cmake? I’ll readily admit I am (very much) a newbie user of cmake. The versions I’ve tested thus far are 3.24.2 which comes bundled with my CLion and 3.25.1; both seem to exhibit the same behaviour. The exception (in ctest) is typically thrown at this line: https://github.com/Kitware/CMake/blob/v3.25.1/Source/CTest/cmCTestRunTest.cxx#L255, which can be seen when running ctest with --debug.

Any ideas?

12

Moderator note: Rather than hijacking a thread, it’s generally better to start a new thread and link to other related threads.

If I had to guess, I’d say what is happening in your case is that your standard output isn’t getting flushed, so you’re losing that output (which would have nothing to do with ctest). Note that exit() does flush all output C streams, whereas I don’t believe abort() does.

1 Like
13

Thanks for this - you’re of course right about not hijacking threads as well. :see_no_evil: I moved this to a separate discussion now and posted a reply there. :+1: