Trying to generate header file using add_custom_command, resulted limping build

Code
,
1

Hi, I’m trying to generate a header file during build.
When I changed the template file(DataTableSchema.h) and try build, ninja only executes custom command and not build a library.
and next time I command build with ninja, ninja detects that generated header file(datatable.hpp) has been modified,
so a library file(lib.cpp) that including it recompiled.

I googled about it a lot, but it seems unusual for people.
https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#how-can-i-generate-a-source-file-during-the-build
all the explanation tells me that they don’t seems experiencing the issue.
and it seems ‘touch’ is necessary to pick ‘depends’ option is it supposed to be necessary?

I’m using cmake with visual studio 2022 integrated environment.

  • VS2022 17.3.4 (i tried with commandline too so i guess it wouldn’t be matter though)
  • cmake version 3.23.22060601-MSVC_2
  • ninja version 1.10.2

Game/knuckles/CMakeLists.txt

add_custom_command(
    COMMAND "py" "${CMAKE_SOURCE_DIR}/../Tools/CodeGenerator/main.py" 
            "-i" "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
            "-o" "${CMAKE_CURRENT_SOURCE_DIR}/include/Knuckles/_generated/datatable.hpp"
    COMMAND cmake -E touch "./include/Knuckles/_generated/datatable.hpp" #without this line depends is not working and custom command runs everytime when we build
    DEPENDS "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
    OUTPUT  "./include/Knuckles/_generated/datatable.hpp"
    COMMENT "Generating code for datatable."
)


add_library(knuckles_lib "./src/lib.cpp" "./include/Knuckles/_generated/datatable.hpp")

# Includes
set(knuckles_lib_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/include"
    CACHE STRING "")
target_include_directories(knuckles_lib PUBLIC "$<BUILD_INTERFACE:${knuckles_lib_INCLUDE_DIR}>"
                                         "$<INSTALL_INTERFACE:./${CMAKE_INSTALL_INCLUDEDIR}>")

#I tried wrapping target trick too but it has no difference
#add_library(knuckles_lib "./src/lib.cpp")
#add_custom_target(run_codegen DEPENDS  "./include/Knuckles/_generated/datatable.hpp")
#add_dependencies(knuckles_lib run_codegen)

also it seems that directory structure does matters?

Game
   - console_app  //executable
       - src
       - CMakeLists.txt //build process usually starts with building the executable
   - knuckles  //lib
       - include/knuckles/_generated //generated header files goes here
       - src
       - CMakeLists.txt  //codegen implemented here
   - CMakeLists.txt //main cmake

Game/kunckles/src/lib.cpp

#include "knuckles/_generated/datatable.hpp"
//below code
2

Solved problem by chaning path explicitly!
I don’t understand what makes this kind of difference…
Please someone explains me, necessary of touch and explicit path!

worked example.

add_custom_command(
    COMMAND "py" "${CMAKE_SOURCE_DIR}/../Tools/CodeGenerator/main.py" 
            "-i" "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
            "-o" "${CMAKE_CURRENT_SOURCE_DIR}/include/Knuckles/_generated/datatable.hpp"
    COMMAND ${CMAKE_COMMAND} -E touch "${CMAKE_CURRENT_SOURCE_DIR}/include/Knuckles/_generated/datatable.hpp" #without this line depends is not working and custom command runs everytime when we build
    DEPENDS "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
    OUTPUT  "${CMAKE_CURRENT_SOURCE_DIR}/include/Knuckles/_generated/datatable.hpp"
    COMMENT "Generating code for datatable."
)

add_library(knuckles_lib "./src/lib.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/include/Knuckles/_generated/datatable.hpp")
3

You should generate the file in the build directory and not in the source directory.

It runs each time because the two file paths don’t match.

4

Could you explain bit more persicely? because before scripts already didn’t generate the header file in every time after I add touch command. but the problem was ninja didn’t catch dependency I think.

It was like that

  1. if no touch on the template file, ninja doesn’t do nothing
  2. if the template file has modified, when I build, ninja runs custom command and generate the header file, but source code that including header file didn’t catch change of header file. so ninja don’t do any compiling.
  3. so after 2, if I try build again, ninja now catches change of header file and build the source code.

I thought that ‘.’ in path represent the current path of that 'CMakeLists.txt". but I guess ninja sees it differently? And somehow changing ‘.’ to the absolute path convince ninja somehow.

5

First, you should try to keep the source directory clean of build artifacts. This allows to have more than one build tree.

Second, the add_custom_command documentation states for OUTPUT that relative path is interpreted relative to current build directory but your command creates in the current source directory. As the specified output file never exists, the rule is executed every time.

The touch command is only needed because your python command does not update the time stamp if it does not create the file with possibly same content. For ninja generator you can alternatively use the BYPRODUCT statement. Then you can use a dummy stamp file as output, the actual output file as byproduct and the file does not change and the compiler does not need to recompile it.

6

Thanks a lot, I now understand what happens with my previous code! :smiley:

I thought that relative path, ‘./path’, in ‘.’ represents current cmake path, but it turns out it could be different per command!

I have following question based on your answer.

  1. generating different file based on build tree sounds nice! but my original intention that putting generating code in ‘include’ directory was,
    (a) I wanted to be in same path(include/knuckles/_generated) when I do install
    (b) I wanted include path won’t changed in my source code
    Is there are solution for that?

  2. I tried with BYPRODUCT stamp technique you mentioned based on answer of below link
    ninja - CMake's add_custom_command doesnt trigger rebuild of library - Stack Overflow
    but pitfall I found is that It is not working if generated file is not pre-existed. Is there are better ways to do that?

7

I found solution for first following question my self, would you think this is correct approach?

file(GLOB_RECURSE codegen_scripts "${CMAKE_SOURCE_DIR}/../Tools/CodeGenerator/*.py")
add_custom_command(
    COMMAND "py" "${CMAKE_SOURCE_DIR}/../Tools/CodeGenerator/main.py" 
            "-i" "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
            "-o" "${CMAKE_BINARY_DIR}/include/knuckles/_generated/datatable.hpp"
    DEPENDS ${codegen_scripts}
    MAIN_DEPENDENCY "${CMAKE_SOURCE_DIR}/../UnrealApp/Source/Knuckles/GameData/DataTableSchema.h"
    OUTPUT  "${CMAKE_BINARY_DIR}/include/knuckles/_generated/datatable.hpp"
    COMMENT "Generating code for datatable."
) #1. CHANGE CMAKE_SOURCE_DIR => CMAKE_BINARY_DIR

# Consturct
add_library(knuckles_lib "./src/lib.cpp" "${CMAKE_BINARY_DIR}/include/knuckles/_generated/datatable.hpp")

# Includes
set(knuckles_lib_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/include" CACHE STRING "")
set(knuckles_lib_GEN_INCLUDE_DIR "${CMAKE_BINARY_DIR}/include" CACHE STRING "")
target_include_directories(knuckles_lib PUBLIC "$<BUILD_INTERFACE:${knuckles_lib_INCLUDE_DIR};${knuckles_lib_GEN_INCLUDE_DIR}>"
                                               "$<INSTALL_INTERFACE:./${CMAKE_INSTALL_INCLUDEDIR}>")

[Main]CMakeLists.txt

install(DIRECTORY ${knuckles_lib_INCLUDE_DIR} DESTINATION "./")
install(DIRECTORY ${knuckles_lib_GEN_INCLUDE_DIR} DESTINATION "./")