try_compile() checks fail with conditional CMAKE_EXECUTABLE_SUFFIX

Usage
#1

I am trying to build a application for VxWorks 7. VxWorks provides a platform (VxWorks.cmake) and toolchain file for CMake that I have adapted a little bit.

The reason for my issue is that VxWorks has two types of executables with two different extensions:

  • RTP (Real Time Process) with an .vxe extension
  • DKM (Downloadable Kernel Module) with an .out extension

The toolchain file provided by VxWorks uses a variable VX_TARGET_TYPE (defaults to RTP) to differentiate between them and adjust compiler flags, include paths, etc. accordingly. I wondered why they did not set variables like CMAKE_<STATIC|SHARED>_LIBRARY_<PREFIX_SUFFIX> or CMAKE_EXECUTABLE_SUFFIX, so I tried to add them. For CMAKE_EXECUTABLE_SUFFIX I used a simple if-else condition to set either .vxe or .out. Thats where things started to break. A simple

check_type_size("int" SIZEOF_INT)

suddenly causes errors like this one:

show/hide error message 
-- Check size of int
CMake Error at C:/Program Files/CMake/share/cmake-3.20/Modules/CheckTypeSize.cmake:139 (try_compile):
  Cannot copy output executable

    ''

  to destination specified by COPY_FILE:

    '<CMAKE_BINARY_DIR>/CMakeFiles/CheckTypeSize/SIZEOF_INT.bin'

  Unable to find the executable at any of:

    <CMAKE_BINARY_DIR>/CMakeFiles/CMakeTmp/cmTC_bb4ba.out
    <CMAKE_BINARY_DIR>/CMakeFiles/CMakeTmp/Debug/cmTC_bb4ba.out
    <CMAKE_BINARY_DIR>/CMakeFiles/CMakeTmp/Development/cmTC_bb4ba.out

Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.20/Modules/CheckTypeSize.cmake:270 (__check_type_size_impl)
  CMakeLists.txt:9 (check_type_size)

I suspect this happens because try_compile() that is used by check_type_size() performs a little subbuild that does not get the value of VX_TARGET_TYPE, but uses the same toolchain and platform file. In my case the toolchain will default VX_TARGET_TYPE to RTP if not set, which is why try_compile() produces a .vxe file, but my build uses DKM and therefore try_compile() will assume a .out extension when searching for the output file here.

When using try_compile() directly, I could use the CMAKE_FLAGS parameter to pass on VX_TARGET_TYPE, but with check_type_size() and possibly other check commands, I can’t do that.

I’m not quite sure if this may be considered a bug in cmCoreTryCompile::FindOutputFile() and if there is some sort of “fix” to achieve the expected behavior. If not, it would be nice to have something like a variable to list other variables that try_compile() should pass on to the subbuild in order to work around this issue. The only workaround I can currently think of is to not set CMAKE_EXECUTABLE_SUFFIX in the platform file and instead adjust the SUFFIX or OUTPUT_NAME target properties manually for every executable target. But this is rather annoying and hard to keep track of in a large project.

Minimal, reproducible example: example.7z (828 Bytes)

No worry, there is no need to buy a VxWorks compiler. It can be reproduced with GCC or some other compiler as well.

#2

Seems like such a variable already exists: CMAKE_TRY_COMPILE_PLATFORM_VARIABLES
Setting it in my platform file (VxWorks.cmake) as follows solves my problem:

list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES "VX_TARGET_TYPE")

Sometimes reading the documentation more closely would make life much easier … :man_facepalming:t2: