In the project dir there is the main.cpp file and a resource.xml file which contains some source code. I am already using DEPENDS_EXPLICIT_ONLY to minimize the set of dependencies.
The custom command transforms the resource file into a cpp file, which is added to the executable source list.
My expectation about the building steps is that the custom command execution together with resource.cpp build step can happen in parallel w.r.t the building of other source files (in this case only main.cpp). However, it seems that there is an additional resource.cpp dependency added to the building step of main.cpp, which prevents this parallelization (on cmake 3.29.3 and trunk).
By generating the Ninja graph output, we can see an order only dependency that is connected to all target sources (main.cpp, resouces.cpp2) build steps, which should not be necessary in our case. We just want a link dependency to the object file of resource.cpp, which is correctly generated.
Setting Policy CMP0154 and an empty file set for the executable target does not seem to solve the issue, as it moves the dependency to the cmake order only private group.
We checked the CMake project source code and it seems that the addition of this dependency cannot be prevented (in cmNinjaTargetGenerator::WriteObjectBuildStatements, ccout and orderOnlyDeps), unless our file is a cxxmodule and specified in the file set.
Am I missing anything? Is there any way to solve this issue?
Do you see the same behavior with CMake 3.27 or older?
@ben.boeckel Some of the comments suggest that there may have been a regression with some of the C++20 modules work, but let’s see what the response to my question just above is first before we conclude anything there.
I think I can only check it on CMake 3.27 because that’s the minimum version that supports DEPENDS_EXPLICIT_ONLY, without which there would be still a global dependency on the custom command dependencies.
Sadly, testing it with CMake 3.27.1 seems to held the same result:
This is a known (to me at least) limitation that I discovered while implementing C++ modules. This MR (that I plan to get back to some day; tracking/discussion issue) would fix it. Basically, because we currently have a grabbag list of sources, CMake has no idea that main.cpp doesn’t #include "resource.cpp" for some reason. By making a FILE_SET that contains sources that we know cannot be #included (like C++ module interfaces are as export module cannot hide behind an #include), we can drop the dependency of other sources in the target on the generated source.
I am glad to know this is a known issue. While trying the new FILE_SET functionality I was wandering why there was not a way to also put options on cpp files too.
I have been reading the issue you linked and it seems that for now there are still some design issue to solve. Just to understand the issue better, I was wondering: if a cpp file is included through an include directive, isn’t the dependency automatically detected by querying the compiler? If yes, why there is the need of a global dependency on the generated files?
Also, the issue of solving this through FILE_SET(s) properties seems to be that it does not allow to configure this behavior per target, but only per file: can’t we change the default behavior so that if a generated file is in the list of target source files, we drop the global order-only dependency at least for that target? Is there any other edge case to cover?
Our use case is really simple, we just need to embed some resources in the code in form of big arrays. The conversion from original files to cpp/h is done through a custom command. Generated cpp files are added to the target source list, then compiled. Right now, the building of any target source file is not allowed until all files have been generated, reducing the potential for parallelization.
Scenario: what is the expected behavior when compiling a source that includes it before the source exists? How is CMake supposed to know that ahead of time?
Basically, CMake ensures anything that could be included is there before compiling potential includers. The compiler detection will mention it and cause recompilation if it changes if it is included, but the only way to handle a new #include line is to have it there in the first place.
A policy could be made…but how to detect it? I suspect it is a policy that cannot warn, so when the policy is set to NEW, projects can start to get non-deterministic build failures based on build tool scheduling. I’m afraid that it’s better to have to inform CMake of this explicitly in some manner (FILE_SET is the most natural, but perhaps a target property could suffice).
I understand now. Basically if the generated file does not exist, than asking the compiler to deduce the header dependencies will fail, because the compiler is not able to determine the path to the dependency, considering that it could come from any of the include paths. Also, the file itself could include more dependencies, which would require it to be generated before the build step. Even if we generate some empty placeholder files for the files which have the GENERATED property in the expected paths, that would not be enough. Am I right?
I understand your concern, even if having a source file which is both included and compiled by the same target seems apparently unusual.
Do you mean like a target property which holds a list of files that are supposed to not be included?
If I can contribute also to write some code, I will be glad to help.
where impl.cpp on its own is “useful” as well.
It’d be a target property that says “all generated sources with a defined LANGUAGE property (headers wouldn’t have one applied) are not needed for compilation of any other TU in the target”, not a list of sources. Per-source properties would be possible too, but sounds tedious to bridge.
