two tier source generation

Code
1

Hi All,

I’m looking for some advice on how declare a library that depends on two custom tools being run.

1. glob the source tree for a list of files
2. run a tool on this list to convert to a slightly different format (one to one relationship)
3. run a second tool on each new file to generate source code (this is a one to many relationship)
4. define a library from the list of generated source code

at the moment I have the first three steps kinda working and putting files in the build tree at build time but I’m getting stuck at the 4th step as the files are not know during the configuration stage.

How would you all structure this?

2

I would probably make a Python/Bash script for performing the first three steps, and that script would be called before running CMake, and then CMake should be able glob the output directory with generated sources. Something like:

$ cd /path/to/project
$ mkdir build && cd $_
$ ../scripts/generate-sources.sh
$ cmake ..

Then the generated files would be there (in ./generated directory, for example) for CMake to glob them on configuration stage.

3

Thanks refit,

I’m trying to set this up so cmake alone controllers build.

So this is what I have now, not sure it’s best practice though.

pseudo code

file(GLOB input_files ...)

foreach(in_file ${input_files)
    add_custom_command(
       OUTPUT ${output_file}
       COMAND...
       DEPENDS ${in_file}
    )
    list(APPEND transformed_files ${output_file}
endforeach()

add_custom_target(transform DEPENDS ${transformed_files}

foreach(in_file ${transformed_files})
    set(output_file1 ...)  # named based on filename component of "in_file"
    set(output_file2 ...)
    set(output_file3 ...)
    add_custom_command(
        OUTPUT ${output_file1} ${output_file2} ${output_file3}
        COMMAND ...
        DEPENDS ${in_file}
    )
    list(APPEND generated_soruce_files ${output_file1} ${output_file2} ${output_file3}
endforeach()

add_custom_target(generated_source DEPENDS ${generated_source_files})

add_dependencies(generated_source transform)

add_library(mylib ${generated_source})

add_dependencies(mylib generate_source)

So this does seem to work but what makes me uneasy is that I’m just hoping that the tool that generates the source code in the second for loop follows the same naming convention consistently.

My second concern is what happens if for some reason the tool generates 2 or 4 files instead of 3.

4

In the first post you said:

I’m getting stuck at the 4th step as the files are not know during the configuration stage

so I assumed you’ve tried that and it didn’t work, but have you actually tried outputting the generated files into some dedicated folder and then just globbing that folder (so it won’t matter how many files are there exactly)?

file(GLOB generated_files path/to/generated/*)

add_library(mylib)
target_sources(mylib ${generated_files})
5

yes I’ve made a bit more progress since my initial post. And yes I have tried globing the result of the 2nd for loop.

The result of globing the path to the generated files is empty during the configuration stage and so the cmake complains that a library was declared with no source files

Can you delay or set a dependency on when glob is executed?

6

The result of globing the path to the generated files is empty during the configuration stage

Yeah, I suspected as much, which is why I suggested doing the generation steps before running CMake.

I do not know if it is possible to make globbing aware of files that will be generated at a later point, so I’ll subscribe to the topic to see what others might suggest.

7

To make this reliable, you’ll just need to know the sources that will be generated. If possible, you can also create a “unity” build source correlated 1-to-1 with the input files that are generated to include all of the generated sources that are relevant. This won’t work if they are C++ module-generating (and probably not if they consume them either), but should be fine for non-module-aware C++.

No. It is even more problematic than globbing your source tree. I’ve posted about it to Reddit recently, but am working on a blog post about it. I’ll try to remember to come back and drop a link when it is ready.

1 Like