I am using cmake 3.20.5 and we have created a quite big build system for windows / Linux and Mac.
One of our library (OBJECT library) includes a big number of object files and when using TARGET_OBJECTS macro on it , from time to time - I can say even rarely, one of the object file in the list is missing a character, causing the build to fail.
This happens quite randomly and is also rare - so difficult up to impossible to supply a reproduction of the problem.
Does it sound familiar to someone, workaround? fix?
The content is always written to a file during the CMake call; during the build this information is only used.
So if the build fails, you can easily check if that character is really missing in the CMake output or not.
Does it happen on the same PC, then it may be simply RAM or disc problems. Other random problems mainly involve “outer” parallelization, like someone (or CI system) is calling the same build 2 times.
Yes this is the generator expression: $<TARGET_OBJECTS:target>.
Following your recommendation I have been looking into the generated files and it looks weird. Let me elaborate and maybe we can drill down to the root cause.
We are using the generation expression to dump the object file list into a file in the following way:
The resulting build log file shows the above command with the right file list. This is true for the .vcxproj generated. But some_file.txt includes the exact same list but one object file name is being missing a character.
It seems to happen on a specific machine and I am not sure it does not happen on others.
Would you say that “${CMAKE_COMMAND} -E echo” may have an issue with very long lists?
Is there an alternative way to dump this list to a file?
The most problems, like “command line too long” or “special unquoted character on command line” are fully reproducible.
And that’s why you should write the file as Scott suggested. This avoid any of such problem in general.
But it’s still not an explanation for missing a single character. If the problem disappears by simply starting the same build a second time (after a clean), then I’m quite sure that it’s a hardware problem (RAM, CPU or …). You can try to do a RAM and CPU Load test (e.g. MemTest86 and Prime95) on that PC.
First thanks a lot for your suggestion above which seems to work.
Somehow file(GENERATE …) does not have this issue.
Although file(GENERATE OUTPUT some_file.txt CONTENT …") is a bit tricky since I got some errors saying:
“Evaluation file to be written multiple times with different content”
I had to play a little with the file name and have $<CONFIG> in it otherwise CMake is not ready to accept. Finally it passed.
Answering to Joseph: the problem with the missing character is consistent but only with specific machines - it seems that “windows Server 2016” is having that issue but other machines do not - so I do not think this is a RAM or hardware issue since if it was, then the issue would not happen with that machine repetitively and at the same place: this problem appears randomly on environment / hardware but it occurs again and again once you have a machine with the issue and at the same place.
Really weird and difficult to imagine a bug with a behavior like this …
Is the missing character always the first one of a file name or directory name in a path? I’m thinking you might be seeing cases where backslashes are used in paths, but CMake is treating them as escapes. If those correspond to recognised escape sequences (\n, \t, \r and so on), that may explain why those particular characters would be missing in the output.
Note that this isn’t describing a bug in CMake, this would typically be a bug in the way you’ve defined the project or provided information to CMake.
@craig.scott But I guess this should be a reproducible problem.
@serge As you recognized now, that you’re writing different contents to the same file, we can come to the other root cause for randomness: parallelization.
Do you build the different configuration in parallel? If yes, the file might be overwritten by one config build while read by the other one.
If that’s really the case, the old way with echo should also be stable as soon as you use distinct file names.
@craig.scott - Sometimes we get …\xxx.ob instead of …\xxx.obj which hits the last character of a specific object file in a long list of object files. Some other times we got …\director_name… instead of …\directory_name… so I am not sure it is located at some special file path name and is related to escape characters.
@Josef - This is interesting and I am not sure we are doing the right thing. This is windows and we build a specific configuration, say Release. The generation phase is done with:
“cmake -H. -Bbuild -DCMAKE_BUILD_TYPE=Release -G “Visual Studio 16 2019” -A Win32”
while the build phase is done with:
“cmake --build build --config Release --target install”
I saw indeed that file(GENERATE OUTPUT “list.$.str”…) creates two files: list.Debug,str and list.Release.txt which surprised me since we have set release above … Could there be a miss in the way we invoke CMake?
We are meeting the same problem that some characters will be missing when using CMake on Windows server 2016 build server, is there any solution found for this problem?
Without any project to reproduce the problem with, no-one is working on this as far as I’m aware. If you can provide a minimal project that does reproduce the problem, please report an issue in the CMake issue tracker and include that minimal project in the bug report. Without that, we can’t really move this forward.
Well, this one was nasty for us.
The problem was that CMake on windows is a multi configuration generator.
It means that it generates the build files for several configurations. This is not a problem on itself, but it means you need to be careful.
By default you have (I think) at least debug and release configurations.
So when you generate a file in your CMake file, for example:
file(GENERATE OUTPUT “list.txt” <some content>)
Since CMake generates both release and debug
then you may have a write in parallel for both configurations to the same file causing possibly a content corruption as the missing character; this happened during the CMake generation stage.
So the solution was to have a different output file name for each configuration, avoiding the potential write race.
If you generate some file during the generation phase such as an object file list for linking, then make sure the file name are different for each configuration. Our solution to the above was: