How to check for incorrect dependencies when using add_custom_command?

Code
1

Recently I have discovered that I have a problem in my CMake file that can lead to a generator for source files being run while the sources are being compiled. I think I know how to fix this instance, but what I am looking for is a way to discover if any other dependency problems are present in my makefiles.

I could try to build many times and vary how many parallel build processes are running, but I feel that is unlikely to discover these problems. Given that with these makefiles certain source files can be generated while they are also being read would suggest that there is a smarter check that can be done (on a dependency tree perhaps?).

Here’s the code with the problem:

execute_process(
    COMMAND ${MY_GENERATOR_EXE} --input ${GENERATOR_INPUT_DIR}/main_generator_input.txt --output ${GENERATOR_OUTPUT_DIR} --listfiles
    OUTPUT_VARIABLE GENERATED_SOURCE_FILES
)

add_custom_command(
    OUTPUT  ${GENERATED_SOURCE_FILES}
    DEPENDS ${GENERATOR_INPUT_DIR}/*.*
    MAIN_DEPENDENCY ${GENERATOR_INPUT_DIR}/main_generator_input.txt
    COMMAND ${MY_GENERATOR_EXE} --input ${GENERATOR_INPUT_DIR}/main_generator_input.txt --output ${GENERATOR_OUTPUT_DIR} 
)
 
add_custom_target(generator_target
    DEPENDS ${GENERATED_SOURCE_FILES}
    SOURCES ${GENERATOR_INPUT_DIR}/secondary_generator_input1.txt
            ${GENERATOR_INPUT_DIR}/secondary_generator_input2.txt
            ${GENERATOR_INPUT_DIR}/main_generator_input.txt
)

add_library(MyLibrary
    ${GENERATED_SOURCE_FILES}
)

I think the fix is adding this:

set_source_files_properties(${GENERATED_SOURCE_FILES} PROPERTIES GENERATED 1)
add_dependencies(MyLibrary generator_target)

Does anyone have ideas on how to look for problems like this?

2

The most fool-proof way is to, for each output (object compile, target, whatever), do:

$ make clean
$ make -j1 $output

If any output is missing dependencies, this will show which ones. Yes, it is a long loop, but you should be able to test a few special instances and assume “others like it” are OK (e.g., if one compilation in a target works and needs outputs generated, others needing a subset of that same output set is also OK).

On to your example:

This is meaningless; globs are not something CMake models here.

Nothing seems to declare that it makes these files, so that seems like something is missing. They’re probably in the source tree; no need here. Other custom commands might want to declare these as dependencies though?

Note that you should declare the input file(s) that matter here as CONFIGURE_DEPENDS as if the file changes in such a way that the set of output files changes, CMake needs to regenerate the DEPENDS.

3

If you’re using Ninja, you could also try its missingdeps tool: The Ninja build system

4

That’s a nifty tool, but seems to require add_custom_command(DEPFILE) to be used where possible.

5

Thank you for your feedback. I tried to incorporate it in this updated version.

  • I no longer specify a glob in the DEPENDS of add_custom_command. Instead I add a file(GLOB) command to create a GENERATOR_INPUT_FILES variable. This also allows me to specify CONFIGURE_DEPENDS as suggested.
  • I also added a dependency on the generator to add_custom_command, which was missing in my original post
  • finally I now have two libraries using the generated sources as an example that shows the type of problem that I am trying to detect. If I forget to add a dependency on the generator_target to one of the libraries (see commented-out line) then there will still be a dependency on the generated source file, but it can cause the generator to be spawned twice at the same time in a parallel build. This type of problem I would not find by building each individual target from a clean workspace.

CMake knows that the custom command produces those files and it knows that two targets need those files. Isn’t this a type of mistake that CMake should be able to detect at configure time? Or should perhaps the dependency be implicit so that a user cannot even make this mistake?

project(CMakeTest)
cmake_minimum_required(VERSION 3.25)

set(MY_GENERATOR_EXE python.exe ${PROJECT_SOURCE_DIR}/generator.py)
set(GENERATOR_INPUT_DIR ${PROJECT_SOURCE_DIR}/generator_input)
set(GENERATOR_OUTPUT_DIR ${PROJECT_SOURCE_DIR}/build/generator_output)

file(GLOB GENERATOR_INPUT_FILES CONFIGURE_DEPENDS ${GENERATOR_INPUT_DIR}/*.txt)

execute_process(
    COMMAND ${MY_GENERATOR_EXE} --input ${GENERATOR_INPUT_DIR}/main_generator_input.txt --output ${GENERATOR_OUTPUT_DIR} --listfiles
    OUTPUT_VARIABLE GENERATED_SOURCE_FILES
    COMMAND_ERROR_IS_FATAL ANY
)

add_custom_command(
    OUTPUT  ${GENERATED_SOURCE_FILES}
    DEPENDS ${GENERATOR_INPUT_FILES} ${PROJECT_SOURCE_DIR}/generator.py
    MAIN_DEPENDENCY ${GENERATOR_INPUT_DIR}/main_generator_input.txt
    COMMAND ${MY_GENERATOR_EXE} --input ${GENERATOR_INPUT_DIR}/main_generator_input.txt --output ${GENERATOR_OUTPUT_DIR}
)
 
add_custom_target(generator_target
    DEPENDS ${GENERATED_SOURCE_FILES}
    SOURCES ${GENERATOR_INPUT_FILES}
)

add_library(MyLibrary
    ${GENERATED_SOURCE_FILES}
)
add_dependencies(MyLibrary generator_target)

add_library(MyLibrary2
    ${GENERATED_SOURCE_FILES}
)
#add_dependencies(MyLibrary2 generator_target)