Make miss dependencies when add_custom_command depends on add_custom_target or add_custom_command

Code
1

Hi,

When in a subdirectory’s CMakeLists.txt, I have:

add_custom_command(
	OUTPUT ${TEST_OUTPUT}
	COMMAND ${CMAKE_COMMAND} -E copy ${TEST_INPUT} ${TEST_OUTPUT}
	DEPENDS ${TEST_INPUT}
)
add_custom_target(gen_test_file DEPENDS ${TEST_OUTPUT})

And in parent CMakeLists.txt:

add_custom_command(
	OUTPUT test_file2.txt
	COMMAND ${CMAKE_COMMAND} -E copy $<TARGET_PROPERTY:gen_test_file,GENERATED_FILE> test_file2.txt
	DEPENDS gen_test_file
)
add_custom_target(gen_test_file2 ALL DEPENDS test_file2.txt)

The gen_test_file dependency is not honored by make and NMake when ${TEST_OUTPUT} is changed.
If I touch the ${TEST_INPUT} file, gen_test_file is rebuilt but not test_file2.txt:

[ 50%] Generating test_file.txt
[ 50%] Built target gen_test_file
[100%] Built target gen_test_file2

I’m expecting that output instead:

[ 50%] Generating test_file.txt
[ 50%] Built target gen_test_file
[100%] Generating test_file2.txt
[100%] Built target gen_test_file2

I’m attaching add_custom_target_dependencies.zip which contains an example.
I’m executing these commands in the unzip directory containing the parent CMakeLists.txt:

mkdir build
cd build
cmake ..
# For the first build, all files are generated, that's OK everything works as expeced
cmake --build .

# Here I will touch the ${TEST_INPUT} file
cmake --build . --target touch_test_input

# Here, only test_file.txt is regenerated, I expect that test_file2.txt to be regenerated too
cmake --build .

How to add a dependency between 2 custom commands in separate CMakeLists.txt so when first is rebuilt, the second is also rebuilt ?

I’m tempted to put ${TEST_OUTPUT} as a dependency of test_file2.txt command, but as said in the documentation for add_custom_command :

If any dependency is an OUTPUT of another custom command in the same directory ( CMakeLists.txt file), CMake automatically brings the other custom command

As I have different CMakeLists.txt here, I shouldn’t do that.

2

In the Ninja generator, it appears as though the gen_test_file is made as order-only. I assume the Makefiles generator is doing the same in its language. That’s what’s causing the build tool to not trigger the dependency between the two. It works if I use DEPENDS $<TARGET_PROPERTY:gen_test_file,GENERATED_FILE> though since the file dependency gets picked up by the tool properly. That does not work in Make though since the file dependency is not available due to ordering. Adding gen_test_file back to the list makes it work there too.

I don’t know how hard it would be to modify CMake semantics to be correct here. It may just end up being a documentation thing if that’s not feasible.

Cc: @brad.king

3

Yes, I found that doing both: DEPENDS gen_test_file $<TARGET_PROPERTY:gen_test_file,GENERATED_FILE> works both in Ninja and make (and nmake).
But it seems not expected to work according to the documentation as the would make a file level dependency across different CMakeLists.txt.

At least, I think Make should be able to implement the dependency, but other generators like Visual studio may not be able to do so as of now.

I’ve tried to visual studio project generator, and it does the same: only gen_test_file is rebuilt when ${TEST_INPUT} is changed, not gen_test_file2.
I’m not sure what would be needed to make it works under visual studio, but it should be possible to add VS project level dependencies I think.

4

Sounds reasonable. Please file an issue for this. I don’t know if it will just mean clarification of docs that both are needed in the general case or whether we can actually fix it somehow (the Ninja generator is generally the most well-behaved with respect to dependencies, so I’m not surprised it performs better here).

5

I think you should be using ${TEST_OUTPUT} as the dependency of test_file2.txt. I don’t follow why you wouldn’t. By itself " DEPENDS gen_test_file" just means that a target dependency will be made. But the command itself isn’t dependent upon anything. It’ll just run the first time to create test_file2.txt and once that’s done it’ll never run again. It probably should be "DEPENDS gen_test_file ${TEST_OUTPUT}. This creates the target dependency and then if ${TEST_OUTPUT} changes it should run again.

It was recently stated again that “Custom targets don’t have outputs”. unable to get target location. The documentation confirms this “The target has no output file”. What is " $<TARGET_PROPERTY:gen_test_file,GENERATED_FILE> supposed to evaluate to? I couldn’t file that property listed in the documentation. What is it supposed to be copying? I understand that certain generators implement stamp files and such that seems to be an implementation detail and not a real “output” to be depended upon.

6

It’s a custom property set in the example project. It’s just an absolute path.

7

Thanks. I had to log into discourse to be able to download the project.

set_target_properties(gen_test_file PROPERTIES GENERATED_FILE ${TEST_OUTPUT})

I think that OP’s expectation is incorrect for the example. For the command to run it needs a file dependency to trigger it to re-run. The target dependency is required to make sure that the targets get processed in the correct order.

BTW if it wasn’t for it being a custom target it would have worked (as the file-level depedency would have been created since executable and library targets create an output). Refer to the DEPENDS field documentation for add_custom_command().

8

In fact, I’ve opened an issue earlier and that was the tought of Brad King:
https://gitlab.kitware.com/cmake/cmake/issues/20481

Note the “in the same directory” part.

All the custom commands that chain dependencies together within a single target need to be defined in the same directory.

And indeed, when putting only a file level dependency, make will fail to find how to make it.
So the fact that it works with both dependencies (target and file) is maybe just an artifact of how make works, it seems not clearly supported by CMake, or intended to work that way as of now (probably because of other generators like XCode or Visual Studio).

But I agree that in my case, a file level dependency is something that would fit.
I use both dependencies currently (DEPENDS gen_test_file ${TEST_OUTPUT}) as a workaround and it works fine with both Make and Ninja (I’m not targeting XCode or Visual Studio).

9

But there isn’t any issue; at least I don’t see one. add_custom_target has no output.
This leaves two options if add_custom_command depends on a add_custom_target.

  1. The commands of add_custom_target are always run because the “dependency” doesn’t exist.
  2. The commands of add_custom_target only run when a “dependency” that physically exists change.

(2) seems to be the correct answer.

And it’s why add_custom_command should work as expected if it was dependent on a executable or library target. Those targets generate an output; and that output can change if the target is rebuilt.

I think you are asking for a change to add_custom_target so that it can produce an output that can be used as a file level dependency as well as an ordering dependency. That’s an enhancement not a defect.

10

Well, while searching internet for Visual Studio custom command dependencies, I found that blog post explaining this exact thing:

It points out this part of add_custom_command for the DEPENDS field:

If the argument is the name of a target (created by the add_custom_target() , add_executable() , or add_library() command) a target-level dependency is created to make sure the target is built before any target using this custom command. Additionally, if the target is an executable or library, a file-level dependency is created to cause the custom command to re-run whenever the target is recompiled.

So in fact, what I have thought of a workaround, is really what CMake seems to be doing under the hood for executable and library targets, and seems maybe expected.

I was not expecting that to be expected given I was thinking that file level dependencies with add_custom_command are only supported within the same CMakeLists.txt.

Then I think this should be handled automatically so when a add_custom_command depends on a add_custom_target, file level dependencies are added to the add_custom_target’s DEPENDS.
I don’t see a use-case against this that would justify to have the current behavior.

This is effectively an enhancement request in that case :slight_smile:

11

this is probably the same than the add_depenencies() issue multiple users come across.
(e.g. How to create file-dependency for all sources of target?)
While this is documented, i think this is something that should be globally fixed maybe via a policy setting.
IMO: When A depends on a target B, you always want both:

  1. A is built after B
  2. A needs rebuild if B needs rebuild.
12

Not necessarily. Sometimes B does work, but determines that no more work needs done (e.g., the VCS information is the same as last time), so version.h is not updated. A should not rebuild in that case.

13

To me, the version.h generation should be a PRE_BUILD command with a BYPRODUCTS version.h instead of a simple target with dependencies, this more intuitively suggests that it will be done only when the build is made.

But according to the current documentation, this would work only with Visual Studio generator as for other generators, PRE_BUILD is really PRE_LINK instead which is unfortunate.

Then maybe all generators should be updated to work the same as Visual Studio regarding PRE_BUILD and changing to complete custom target dependencies with file dependencies would be doable without feature regression.

14

No, I want it always to build so that it is always up-to-date when any artifact is updated. I can’t add a static dependency on every file that affects the VCS revision/dirty status, so it is an “always run” rule that short circuits if nothing has changed.

15

Ok fair enough.

If I want to create cmake function that mimick the behavior of adding both file and target dependencies between custom targets, can I retrieve add_custom_target's DEPENDS value with something like get_target_property or generator expressions ?
I didn’t found anything in cmake-properties(7) that would allow that.

So I could create say my_add_custom_target() that would do the same as add_custom_target, but for each DEPENDS to an utility target, retrieve its DEPENDS that are files.
Then call the real add_custom_target with DEPENDS equal to my_add_custom_target's DEPENDS + retrieved file depends from utility targets.

I hope $<TYPE:custom_target> would return UTILITY in that case (or an clue that it isn’t a normal target, maybe the absence of that property is sufficient).

I will try to experiment, but I hope this is possible to abstract these dependencies.

16

I’ve checked and a custom target does have its TYPE properties set to UTILITY.
But there seems to be no way to retrieve its DEPENDS.
So probably no way to make my own function.

Is adding new commands or options to have both target and file dependencies set as they would do for other target’s type a welcomed addition to CMake ?

This would simplify cases where the custom target (or whatever name is used instead to avoid confusion with existing behavior) is used to call a program that use a file to generate another (as a compiler is actually doing).

And avoid blog post like this to explain why dependencies between custom target doesn’t work the same way as modern CMake with executable and libraries :slight_smile: .

I see that UseJava.cmake could benefit from this, if one is trying to make 2 jar with 2 add_jar commands:

add_jar(B1 D.java GENERATE_NATIVE_HEADERS D1-native)
add_jar(E1 E.java GENERATE_NATIVE_HEADERS E1-native)
add_dependencies (E1 B1)

This won’t work as expected, E1 will not depend on the file generated by B1.
If B1 is modified, E1 won’t be rebuilt (with ninja).
add_dependencies will only add target level dependencies. (As add_jar will use add_custom_target internally).

17

I think I found a way to that my_add_custom_command that manage this and store the dependencies in a custom property so it can be retrieved by targets that depend on it.
But that require some boilerplate code to forward argument from the function to the underlying add_custom_command.
So the my_add_custom_command function iterate its DEPENDS argument and try to find if there are targets that were created with my_add_custom_command too. For these, a file dependency is added to the custom command.
Not perfect, but it should work (I hope :slight_smile:)