recommended workaround for parallel make executes command multiple times does not work with CMP0071

Code
1

Hi!

I have custom command to create C++ sources to be compiled. One simple example is having part of the sources in a tarball (there are very specific requirements, I know that this is not normal), or have them generated by a script. A common solution / workaround [1] [2] [3] is using add_custom_target().

Unfortunately, with CMP0071 NEW this does not work anymore. I created a small example demonstrating the issue https://gitlab.com/inorton/cmake-custom-concurrency/-/issues/2 which can be found here [new users can only post 2 links]

In a Makefile I could use “.NOTPARALLEL” or create some “stamp file” target (which possibly recursively invokes make), but I don’t know how to best do this from cmake.

Any suggestions?

[1] https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
[2] [new users can only post 2 links]
[3] [new users can only post 2 links]

2

I think add_custom_command may work better here. Have you tried using it to expand the tarball with an appropriate OUTPUT list of files? That might avoid duplication (the Makefiles generator is a bit weird here due to…historical implementation details).

3

Hi,

thank you for your quick reply.

Yes, the add_custom_command() has the files listed as OUTPUT. For my project it helped to use set_property(SOURCE ${all_the_files_here} PROPERTY SKIP_AUTOMOC ON), but for the “minimal reproducing example” (https://gitlab.com/sdettmer/cmake-custom-concurrency/-/commits/issue_2_demo) it does not help (see associated issue for this).

So “it works for me” (because I don’t need CMP0071 here, just wanted to avoid the warning), but apparently it works just by accident and I’m afraid this will catch me up later…

The example project is very small, maybe someone could see if it could be fixed and share the solution :slight_smile:

4

Policies are not something you “choose” to keep the old behavior for. CMake changed its behavior for a reason and the new way is the intended way it should work, so getting the project to work with CMP0071 is important in that context.

I might be able to take a look later, but no guarantees.

Cc: @brad.king @craig.scott

5

Yes I see, that’s why I enabled it (“NEW”). I didn’t even test, but I assume using “OLD” would help, but as you say, just for a short time and it would be against the right direction.

I wonder why I’m first reporting this, is is exotic to use both automoc and source code generation?

6

No, it’s not uncommon (e.g., ParaView uses it extensively). Something must be different in your case as compared to the way ParaView is using it though.

ETA: Though ParaView’s use case involves configure_file, not add_custom_command, so that’s likely the big difference.

7

Any ideas anyone?

From time to time our Jenkins jobs (automatic build jobs) fail and I still have no idea for a good workaround. I have the fear that in some constellation it suddenly could happen always and we would have a software state that cannot be built (we do need escrow and so have to be functionally reproducible).

Any suggestions what I could do?

Should I somehow try to migrate to configure_file instead of add_custom_command? @ben.boeckel could you give me some hints, if so? Like: do I need to enter each file in configure_file? How do I generate them at all? Should I somehow untar them before? Or can I somehow add manual dependencies?
Problem is, that currently in prod code it happens rarely (but in the linked example, it happens always).

8

I don’t know if it is related or should matter in this case, but I noticed in your minimal example that you don’t specify paths to the sources. The OUTPUT of add_custom_command() is assumed to be relative to the build directory. The sources given to add_library() are assumed to be relative to the source directory. BUT there is some wierd logic that will find them relative to the build directory instead if they are not in the source directory. I’m not 100% convinced that we always set up the dependencies right when this fallback to the relative-to-build-directory case is involved. It might just be my paranoia, but worth exploring if you’ve run out of things to investigate and are at a road block.

9

Thanks so much for your time and help!
Indeed the file list of OUTPUT was not correct in the example. I fixed it and it seems to work now! I tried both fileX.cpp and ${CMAKE_CURRENT_BINARY_DIR}/fileX.cpp style (consistently, i.e. same in all places) and found both working. So now the minimum example seem to work fine!
Unfortunately this is different in my real world situation. It already was using ${CMAKE_CURRENT_BINARY_DIR} and I see that are files all absolute, correct paths (I added a check if each file exists and they do). All sources are listed in OUTPUT (with absolute paths). I don’t know why it behaves differently. I will try to re-create the real-world make file from scratch to see if there is some hidden mistake or maybe another dependency has some effect.

10

I tried debugging the Makefiles, but it is very complex. I see that both the “example case” and in my “real world situation” the “generator targets” exist only once (for the very first souce file listed); all other generated files are targets with rules that essentially contain a cmake -E touch_nocreate. I think this shows that cmake got the right idea about what I’m trying to archive.

Interestingly, although in CMakefile.txt I used SOURCE files with absolute path like

set (XYZ_SOURCES
 ${CMAKE_CURRENT_BINARY_DIR}/path/test1.cpp
 ${CMAKE_CURRENT_BINARY_DIR}/path/test1.h
 ${CMAKE_CURRENT_BINARY_DIR}/path/test2.cpp
 ${CMAKE_CURRENT_BINARY_DIR}/path/test2.h
)

and verified with an echo in add_custom_command that the paths are really absolute, in generated Makefiles (such as CMakeFiles/test1.dir/build.make), they are relative:

libtestlib/test1.cpp: /absolute/path/dependency.tgz
        @$(CMAKE_COMMAND) -E cmake_echo_color ..... "Generating sources..."
        ...the real generator / untar rules...

libtestlib/path/test2.h: libtestlib/path/test1.cpp
        @$(CMAKE_COMMAND) -E touch_nocreate libtestlib/path/test2.h

If I understand correctly, for each target a make sub process is created, so when building 4 libs in parallel, 4 make instances start.

Question 1

Each subprocess needs the “generator” for only one file due to the generated rules - but I think the parallel make processes do not know about each other, so how could they know that only one of it actually needs to call the generator at all? That means, how knows make in process #2 that the other make in process #1 is already starting the file generation?

Question 2

I have

set_property(
    SOURCE 
      ${ALL_SOURCE_FILES}
    PROPERTY
      SKIP_AUTOMOC ON
)

in CMakefile.txt, but in the console output of make, I see:

[  0%] Automatic MOC for target xxx

is this expected or is my SKIP_AUTOMOC wrong?

11

That may still be right if those sources are not in the set. If you want to disable it on the target, you can set the target AUTOMOC property to 0.