DEPFILE with Makefile generators

Hi!

I’m writing an add_custom_command in cmake 3.21.2. My command successfully generates what looks to me like a .d format file with its dependencies. I specify this file as DEPFILE in my add_custom_command. I was expecting make to read this file at some point. make doesn’t seem to pay attention to the dependencies expressed there. I stashed an $(error message) at the end of the file, and make doesn’t abort when I run it, so it’s not reading that $(error) line. How can I debug what I’m doing wrong?

Thanks!!!

If you can attach a minimal project which reproduces your problem, it will be more likely someone will be able to help you out. The specific details of what you’re trying to do will likely be important.

Hey!

I’m sorry; I posted my question during a frustrating time. I apologize for its vagueness. Thank you for replying anyway!

I think my specific questions is “Is it true that, in the context of Unix Makefile generators what I specify in DEPENDS when doing add_custom_command is passed as an include to make?” I now know that the answer is “no.” Here is how I verified this:

etobis@galadriel:~/cmake_example$ cat CMakeLists.txt
cmake_minimum_required(VERSION 3.20)

project(example)

add_executable(generated main.c generated_file.c)

add_custom_command(OUTPUT generated_file.c
                   COMMAND echo 'static const int a = 2\;' > generated_file.c
                   COMMAND cp ../depfile.d generated_file.d
                   DEPFILE generated_file.d)
etobis@galadriel:~/cmake_example$ cat depfile.d
$(error aborting)
generated_file.c: /home/etobis/cmake_example/depfile.d
etobis@galadriel:~/cmake_example$ cat main.c
int main(int argc, char * argv[])
{
        return 0;
}
etobis@galadriel:~/cmake_example$ mkdir build && cd build && ~/cmake-3.21.3-linux-x86_64/bin/cmake --version && ~/cmake-3.21.3-linux-x86_64/bin/cmake .. && make && make && make
cmake version 3.21.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/etobis/cmake_example/build
[ 25%] Generating generated_file.c
Consolidate compiler generated dependencies of target generated
[ 50%] Building C object CMakeFiles/generated.dir/main.c.o
[ 75%] Building C object CMakeFiles/generated.dir/generated_file.c.o
[100%] Linking C executable generated
[100%] Built target generated
Consolidate compiler generated dependencies of target generated
[100%] Built target generated
[100%] Built target generated
etobis@galadriel:~/cmake_example/build$ touch ../depfile.d # -> I expected this to trigger a rebuild
etobis@galadriel:~/cmake_example/build$ make
[100%] Built target generated
etobis@galadriel:~/cmake_example/build$ cd .. 
etobis@galadriel:~/cmake_example$ vi depfile.d
etobis@galadriel:~/cmake_example$ cat depfile.d
generated_file.c: /home/etobis/cmake_example/depfile.d
etobis@galadriel:~/cmake_example$ rm build/ -rf
etobis@galadriel:~/cmake_example$ mkdir build && cd build && ~/cmake-3.21.3-linux-x86_64/bin/cmake --version && ~/cmake-3.21.3-linux-x86_64/bin/cmake .. && make && make && make
cmake version 3.21.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/etobis/cmake_example/build
[ 25%] Generating generated_file.c
Consolidate compiler generated dependencies of target generated
[ 50%] Building C object CMakeFiles/generated.dir/main.c.o
[ 75%] Building C object CMakeFiles/generated.dir/generated_file.c.o
[100%] Linking C executable generated
[100%] Built target generated
Consolidate compiler generated dependencies of target generated
[100%] Built target generated
[100%] Built target generated
etobis@galadriel:~/cmake_example/build$ touch ../depfile.d
etobis@galadriel:~/cmake_example/build$ make
[ 25%] Generating generated_file.c
Consolidate compiler generated dependencies of target generated
[ 50%] Building C object CMakeFiles/generated.dir/generated_file.c.o
[ 75%] Linking C executable generated
[100%] Built target generated

I can see from the example above that the inclusion of $(error aborting) somehow causes the system to not process that depfile properly, but not in the way I would expect (i.e., have make say abort with the error message “aborting”). When I remove that line, the file is processed correctly. I’ve tried looking at the cmake source code, but I couldn’t find where these files are being processed. The existence of this line

./CMakeFiles/generated.dir/build.make:  cd /home/etobis/cmake_example/build && $(CMAKE_COMMAND) -E cmake_depends "Unix Makefiles" /home/etobis/cmake_example /home/etobis/cmake_example /home/etobis/cmake_example/build /home/etobis/cmake_example/build /home/etobis/cmake_example/build/CMakeFiles/generated.dir/DependInfo.cmake --color=$(COLOR)

suggests to me that the .d file is processed by cmake in some way, but not fed directly to make. I’ll have to keep digging to see why my big project isn’t working as expected. I was trying to use $(error) statements to confirm that the .d files were being read by make. Alas, it looks like that’s not what should happen.

Thanks again!!

I think @marc.chevrier implemented depfile support for Makefile generators, which first appeared in CMake 3.20. Marc, any idea what might be happening here?

Thank you for your help with this.

To be extra clear: I’m not claiming there’s any wrong behavior or bug in cmake. I just assumed that passing a .d file that causes make to blow up would cause make to blow up. That may very well be an incorrect assumption on my part.

Indeed, he appears to have pushed that change: https://gitlab.kitware.com/cmake/cmake/-/commit/cfd8a5ac1f443725342517ddbaee51692d8d0324 A cursory reading of that change suggests that cmake is opening the .d file and trying to fix paths, as opposed to telling make to -include it. Perhaps when parsing .d files CMake ignores things like $(error). Now that I know where this is happening, I’ll point my debugger there when I’m running cmake to confirm.

Thanks again!

I think there is some misunderstanding about how dependency files work.

A dependency file itself does not cause a rebuild, so it doesn’t matter if this file is touched. The file only adds other files as dependencies, and if one of those is changed, there will be a rebuild.
Example: the make knows that it needs the .c file to create the .o file. After the first compilation the .d generated by the compiler file tells which .h were included to build that .o. So on the next make call it knows which additional files needs to be checked to decide if a rebuild of .o is needed.

Adding the .d file as dependency usually makes no sense and probably causes problems on Ninja, where the file is deleted after reading it.
In a plain makefile you may inject any commands by such a .d file, but it’s purpose is to only specify dependencies. So other build tools may ignore or even fail on such lines.

Thanks, @jtxa, for your reply!

I’m sure I misunderstand something, but I’m not sure it’s exactly what you said. In my example, the DEPFILE is generated_file.d That’s what I tell CMake with the DEPFILE clause. I don’t expect that touching that file will trigger a build. The file I’m touching is called depfile.d. Granted, the name would lead one to believe that’s “a depfile.” Indeed, its contents are in “depfile format,” but in this setup it’s being used just as a dependency of generate_file.c

I’m a CMake novice, but I have some experience with make. I’m used to -include’ing .d files, and I was expecting an $(error) statement in there to cause make to blow up. I see now that was not a valid expectaction.

In case someone knows, my actual problem is that I’m working on a big project, I have a custom_command, I put a DEPFILE in it, and things are not behaving the way I expect them to behave.

  1. How can I confirm that the DEPFILE is being read? By make, CMake, or whatever is supposed to read it. To be clear, the depfile I’m using has one target and many dependencies listed for that target. It looks exactly like the output of a gcc -M run.
  2. Once I know the file is being read, how can I tell that it’s being processed correctly? That is, how can I tell that the dependencies in the depfile are being “attached” to the target? That should let me check that, for example, the paths I used are correct.

Thank you!

Sorry, I got confused by the filename depfile.d which is hand-written file just for this example. I now reproduced your steps.

Everything but the $(error aborting) looks fine. Please have a look at the generated compiler_depend.make, it contains these two rules:

$$(error:

aborting):

If your depfile only contains the dependency in the first line, it should be like this:

generated_file.c: ../depfile.d

In CMake v3.21.0-rc2 both parts are contained, but later CMake version seems to parse only the first line of the dependency file. Personally I wouldn’t consider this a bug, because a .d is there to declare dependencies and is not a generic makefile.

Again, sorry for the confusing name of that file. I agree that this is not a bug in CMake, only in my expectations. Knowing that I can only expect it to read the first line helps tremendously.

Thank you!!

The DEPFILE argument of add_custom_command expects a file containing dependencies in the following format (format compatible with what gcc compiler generates for dependencies):

<target> : <dependency>+

<target> and <dependency> must be absolute paths or paths relative to CMAKE_CURRENT_BINARY_DIR. Backslash can be used to specify dependencies on multiple lines:

<target> : <dependency1> \
<dependency2>

This format is supported regardless of the generator used. CMake parse this dependency file, paths are normalized (and reformatted, depending of build tool used: nmake requires a different format than GNU Make) and, in case of Make generators, dependencies are aggregated in various files compiler_depend.make.

So, the depfile specified in add_custom_command() is never read by build tool, and it is an error to inject any make tool construct (like $(error...)).

Thank you, @marc.chevrier, for this thorough answer. One more question, since I’ve got you here. Is it OK to have multiple of these entries? From what you write I assume that

<target1> <target2>: <dependency>+

is not valid, but is

<target1>: <dependency>+
<target2>: <dependency>+

valid?

Your explanation for why this is not fed directly to the tool is very clear.

Thanks again!

The first form is also valid. Multiple targets can be specified before the :.