Behavior of where CMP0118's value is used is ambiguous

Development
1

I’m hitting what seems to me like some unexpected behavior of the CMP0118 policy (found as part of trying to solve QTBUG-95200). My straw man test project is creating a library in the top level CMakeLists.txt file and doing something in a subdirectory that adds a generated .cpp file to it. After some experimentation, I’ve found that the global visibility of the source file’s GENERATED property depends on the value of the CMP0118 policy at the end of the directory scope in which the target was created. That’s problematic for SDK’s like Qt which generate code but are not in control of the policy setting at the end of the directory scope of the targets from user projects. What I’m finding is that this policy setting is essentially useless in these situations because I’m finding that I’m still having to manually propagate the GENERATED source file property to the target’s scope in order to guarantee the target knows the file is generated. This seems like a broken promise of what CMP0118 was supposed to do. I don’t know if there’s much we can do about it now though, since this policy was introduced in CMake 3.20.

I also note that the behavior of when the CMP0118 policy takes effect is not mentioned at all in the CMP0118 policy docs. The actual behavior was about third on the list of what I thought might be the way it works. The docs need to be updated to clearly spell out how and when the CMP0118 policy is used.

I also note further confusion as discussed in Unexpected behavior of the GENERATED source file property and CMP0118, which also highlights other scenarios where the intuitively expected behavior doesn’t match the actual behavior.

I’ve created this thread for initial discussion. It’s not clear whether there are bugs in the implementation, whether behavior changes are needed, or both. Regardless, doc updates will be required, but consensus is needed on what the behavior should actually be first.

@dbahadir @brad.king

2

The documentation could be updated to clarify when the policy setting is needed, but I don’t think we can change it now. Over time the policy will be set to NEW in more and more places and the problem will go away.

The policy mechanism and scoping is designed to prevent libraries from forcing policy settings on their consumers. Otherwise updating a dependency could break an otherwise working build. Projects consuming Qt need to be updated to set the policy.

3

I think, there is a bug in the implementation.

I played around with the example from Unexpected behavior of the GENERATED source file property and CMP0118 and must say that I am really confused about the result.

Foremost, I thought a call to get_source_file_property for the GENERATED property would no longer result in a NOTFOUND but a 0 instead.

However, I think I know where at least some of the confusion might come from.

I originally thought that there will only every be one instance of a cmSourceFile for a specific file-path. (And the unit-tests for CMP0118 that I implemented probably reflect that assumption.) But that seems not to be true!

This is how I think to have understood it now:

There is either only one instance per processed CMakeLists.txt, or a common instance only exists for the CMakeLists.txt file in which it was first created and all its sub-CMakeLists.txt files. (When trying to access the same file afterwards from a parent-CMakeLists.txt file a new instance will probably be created.)

In general, that should not be a problem, because if any instance of cmSourceFile for a specific file-path was marked as GENERATED, that information is stored in the cmGlobalGenerator. So, each instance only needs to check there. (Although some instances might check too early…)

However, the problem is that for checking the cmSourceFile instance needs to know its full-path, which therefore first needs to be resolved.
But trying to resolve the full-path for non-generated files might be wrong in that situation! (Especially, as the logic for resolving the full-path and supporting missing extensions etc. is quite hard to understand. Is all that really needed? Isn’t it already deprecated for quite some time now?)
Then again, generated files will (probably) always be constructed with a full path so that their full-path could be resolved immediately. (Is that correct @brad.king ?)

So, possibly the solution would be to extend the cmSourceFile::GetIsGenerated function to additionally check the global store (and therefore at least temporarily resolve the full-path).
I tried it and the GENERATED property was visible in more situations. However, that was only a quick and dirty test. (And the OLD behavior would possibly become even more like NEW than it became already through the original implementation.)

4

There is either only one instance per processed CMakeLists.txt, or a common instance only exists for the CMakeLists.txt file in which it was first created and all its sub-CMakeLists.txt files.

For a given source file, there is one cmSourceFile instance for each directory (CMakeLists.txt file) that needs one. There is no inheritance: cmMakefile::InitializeFromParent does not copy the SourceFiles object.

the logic for resolving the full-path and supporting missing extensions etc. is quite hard to understand. Is all that really needed? Isn’t it already deprecated for quite some time now?

It is still needed for many years to come. Policy CMP0115 was recently added.

generated files will (probably) always be constructed with a full path so that their full-path could be resolved immediately.

Yes. cmSourceFileLocationKind::Known handles this.

However, the directory in which we are checking “is this generated?” doesn’t know that it is generated or whether the absolute path is fully known.

5

That isn’t always the case. When functions are defined, it is the policy settings at the time the functions are defined that get used, at least for most policies. That’s because the function authors may need to ensure that their code uses particular policy settings. We use this even in CMake’s own modules quite often. This behavior with the GENERATED property is an example where we can’t do that. The key difficulty is that the library code can not only not control the policy setting, it also can’t know what setting will be used. That basically means code has to be written in such a way that it works for both OLD and NEW settings, which basically means… avoid writing any code that could be affected by the policy. In a nutshell, while the policy can be used directly by projects, any packages that provide commands for other projects to use will never be able to assume the GENERATED property is visible globally and have to manually copy that property to the directory scope(s) they need it to be seen in, and it isn’t always possible to know what those scopes are. That’s a real bummer, because it is not unusual for such commands to be the place where such generated files are defined.

6

5 cents from me, that could be helpful to someone while debugging the related issues:

cmake_minimum_required(VERSION 3.16)

project(mytest LANGUAGES CXX)

add_executable(mytest main.cpp)

cmake_policy(PUSH)
cmake_policy(SET CMP0118 OLD)
set_source_files_properties(main.cpp PROPERTIES GENERATED TRUE)
get_source_file_property(is_generated main.cpp GENERATED)
message("is_generated ${is_generated}")
cmake_policy(POP)

cmake_policy(PUSH)
cmake_policy(SET CMP0118 NEW)
get_source_file_property(is_generated main.cpp GENERATED)
message("is_generated ${is_generated}")
cmake_policy(POP)

The above code prints:

is_generated 1
is_generated 0

that doesn’t look right. For sure we should keep policies consistent across the places where we rely on them. But still I didn’t expect that get_source_file_property returns opposite values in the same directory scope.

7

Well, that certainly seems worth a test case (even if just to keep this…oddity compatible. Another new policy may be able to fix it though.

Do you mind filing an issue with this case?