Add Build Dependency for Configuration Files Consumed by Code Generation Script

Usage
,
1

Hey,

I’ve tried searching the forums and internet for this, but solutions I’ve tried haven’t worked.

Our codebase uses a Python script that generates some final stages of C++. This Python script is called by using

add_custom_command(
    OUTPUT custom_file.cpp
    COMMAND ... GENERATOR_EXPRESSION ...
)

This files is then compiled as an object library, and then immediately linked to the final executable.
The file generation script itself has files that it reads that live inside certain sub-libraries. So, if a config.json file is found in a sub-library then this script reads properties from that file to do the code generation.

The problem I’m running into right now, is that I need to somehow express that a change in one of these files – which can (hopefully) occur – after configuration affects the build. So, CMake should reconfigure itself (which is fine) and force a rebuild of relevant binaries. A single config.json file changing means that all targets that use this generated code should be recompiled. These config.json files are essentially configuration files for the Python Code Generator.

I used the “hack” of using a stamp file with a custom command, and then adding dependencies to try and do this. I can get CMake to notice if a file has been modified, but it only rebuilds that file, and doesn’t propagate. This is a rough, slimmed down flow of what’s going on.

add_library(invalidate_lib INTERFACE)

add_library(obj_lib_0 OBJECT ...) # has config.json
add_library(obj_lib_1 OBJECT ...) # has config.json
add_library(my_lib )
target_link_libraries(my_lib PRIVATE obj_lib_0 obj_lib_1)

add_executable(my_app ...)
generate_code(OUTPUT file.cpp) # this is an add_custom_command
add_library(my_app_gen OUTPUT file.cpp)
target_link_libraries(my_app PRIVATE my_app_gen invalidate_lib)
add_dependency(my_app invalidate_lib)

set(stamp_file "f.stamp")
add_custom_command(OUTPUT ${stamp_file}
    DEPENDS config.json
    COMMAND ${CMAKE_COMMAND} -E touch ${stamp_file}
)

add_custom_target(${Name}_build_stamp ALL DEPENDS ${stamp_file})
add_dependencies(invalidate_lib ${Name}_build_stamp)

What I was hoping to express was that this generate file won’t be in-sync with the config.json files unless it is regenerated. But, my_app_gen isn’t getting regenerated. What feels extra difficult, is that these config.json files live in sub-libraries, so grabbing all of them without globbing might not be easy, so I was trying to propagate this through the use of invalidate_lib.

How might I be able to express this dependency? I’m fine if I need to invoke CMake to reconfigure itself somehow. Meaning, I don’t have to try and force this to be done at build-time. But, I’m open to all solutions.

I’m trying to figure out how to summarize this well.
I have configuration files that affects multiple automatically generated libraries. The Code Generator itself is called with add_custom_command since it’s a Python script. I need to figure out how to get these files re-generated with a configuration file changes.

2

Based on a quick look at your cmake - What I think is your problem, is that when json file changes, it triggers the stamp file and stamp target to be invalidated (ok) - and this in turn triggers the invalidate_lib to be marked out of date - BUT - just because the invalidate_lib is out of date, does not trigger the regeneration of your file/s - you need to make sure that the target that runs the

generate_code(OUTPUT file.cpp)

DEPENDS on the invalidate stamp file/json directly.

The stamp file seems unnecessary too - why not just depend on the json directly. You can do a search of json files, put them in a list and then just DEPENDS ${list_of_json} in the right places? no?

eek! Battery flat. Cannot read/post more …

3

So, I think I’ve tried depending on the json files directly, but I might have been doing it wrong.
How would you suggest adding the json files directly? I’m definitely open to that solution and it would simplify this.

You described the problem well. I spent a while trying to figure out how to rephrase this well.

4

what I tried to say was in essence that this

is fine. it says, rebuild my_app when invalidate has changed. But it doesn’t say, rerun my custom target rule to regenerate the files from the json stuff.

why not …

generate_code(OUTPUT file.cpp DEPENDENCY config.json) # or a list of json files

and inside the generate_code function

add_custom_command(
    DEPENDS ${THE_DEPENDENCY}
    OUTPUT custom_file.cpp
    COMMAND ... GENERATOR_EXPRESSION ...
)

this ties your regeneration directly to the json file, I’m not sure you need the stamp file stuff. It really ought to work, I’m doing similra things all over the place - (but the above is from memory) …

5

Revisiting this stuff now. I’ll be trying it out. I wish there was a more direct way of doing this. It would be very nice if I could just say add_dependency(my_app config.json)