# Proposal: Dependency pre-scanner for generated includes

**URL:** <https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558>\
**Category:** Development\
**Tags:** gen:ninja\
**Created:** [June 12, 2021, 2:25pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558 "2021-06-12T14:25:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 12, 2021, 2:25pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/1 "2021-06-12T14:25:28Z")

</div>

Hello. This is my first post.

# Abstract

Specify generated files and their action, instead of add\_dependencies()

# How we are doing now

Assume just lib2.c depends on the generated header.

```auto
# CMakeLists.txt
cmake_minimum_required(VERSION 3.13)

add_subdirectory(inc)

add_library(lib1 lib1.c)

add_library(lib2 lib2.c) # depends on generated header
target_include_directories(lib2 PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/inc)
add_dependencies(lib2 headergen)

add_library(lib12 lib12.c)
target_link_libraries(lib12 INTERFACE lib1 lib2)

add_executable(main main.c)
target_link_libraries(main PRIVATE lib12)

```

- add\_dependencies(lib2 _headergen_) is essential.
- _lib12_ and _main_ depend on _headergen_ as order-only. It is not intended.

```auto
# inc/CMakeLists.txt
set(template_file ${CMAKE_CURRENT_SOURCE_DIR}/template.h)
set(generated_file ${CMAKE_CURRENT_BINARY_DIR}/generated.h)

add_custom_command(
  OUTPUT ${generated_file}
  COMMAND ${CMAKE_COMMAND} -E copy_if_different
  ${template_file}
  ${generated_file}
  DEPENDS ${template_file}
  )

add_custom_target(headergen DEPENDS ${generated_file})

```

# Proposal

Add an ability to specify _headergen_ as “dependency of file generator”  
It may be similar to include\_directories thing rather than order-only dependency. It may be transitive.

## For example

If a new command would be introduced;

```auto
add_library(lib2 lib2.c)
target_include_directories(lib2 PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/inc)
target_include_generated_files(lib2 PRIVATE headergen) # May be custom target or generated file

```

If include\_directories() could distinguish directories and targets;

```auto
add_library(lib2 lib2.c)
target_include_directories(lib2 PRIVATE 
  ${CMAKE_CURRENT_BINARY_DIR}/inc
  headergen # May be custom target or generated file
  )

```

## Implementation

- If a generator is capable of dyndep, _headergen_ shall be converted to “a dyndep scanner to discover dependencies”. _headergen_ will be invoked when a scanner discovered dependency to _generated.h_
  - A generator is responsible to extract file list to pass to a scanner as “list of missing files to be generated”

- For other generators, _headergen_ may be treated as local order-only dependencies, or global order-only dependencies (like add\_dependencies)

### Thoughts to granularity of scanning

It seems already-implemented dyndep facility emits per-file scanner. I think it is too fine-granularly. Per-target scanner would satisfy.

## Requirements to the scanner

- The scanner should traverse include path with missing file list.
- The scanner should not miss “actually referenced missing files”.
  - Redundant files may be acceptable. “The scanner reports them but they are actually not referenced.” Gross scanner is acceptable (like cmake\_depends) Of course, this is suboptimal.
  - If the scanner is not a gross scanner, it may stop when it met unknown missing files.

- The scanner may omit reporting static files. Just missing files may be reported. DEPFILE (in compilation) shall cover them.
  - That said, the scanner should traverse all include files. Just may omit emission.

- The scanner should emit its DEPFILE for the scanner itself. It should contain all scanned files, even if they were not emitted.

Makefiles generator’s “cmake\_depends” could be diverted to this, but I guess it’d be hard to use it from Ninja generator.  
For now, an external scanner may be specified.

### Future of the scanner

The scanner may be capable of scanning also pre-compiled headers, like Clang’s “modules cache”  
Then, an external scanner, for example clang-scan-deps, shall be specified.

# Background and history

I have been experimenting ninja dyndep for weeks. I achieved highly parallelized build.ninja for Clang/LLVM tree.  
See, “Clang can be built within one minute”, [https://twitter.com/chapuni/status/1401519362058555393](https://twitter.com/chapuni/status/1401519362058555393)

For it, I implemented in CMake, “automatically discover add\_dependencies() to generated header file and convert them to dyndep”

> **[GitHub - chapuni/CMake at discover\_dyndep/trunk](https://github.com/chapuni/CMake/tree/discover_dyndep/trunk)**
>
> discover\_dyndep/trunk

It works fine with LLVM tree but I don’t think it would be suitable to generic use.

- I don’t think all projects would be ready to discover\_dyndep. I guess some of them would be immature.
  - Introducing a new policy might be an option.

- It was hard to dissolve cmake\_object\_order\_depends chain.
- Dissolving add\_dependencies() would not help cutting unintended dependencies.

I concluded I should introduce more simple scheme. (see proposal above)

Thank you.

---

<div class="post-metadata">

**Author:** ![marc.chevrier](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/m/ecb155/32.png) [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)\
**Post date:** [June 12, 2021, 2:36pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/2 "2021-06-12T14:36:17Z")

</div>

Starting from `3.20`, `Makefiles` generators, as `Ninja`, use the compiler itself for C, C++, CUDA, etc… to compute the dependencies so, as soon the header file is included in some source file, dependencies will be correctly computed without any special treatment.

Considering the previous remark, is there any added value to create `target_include_generated_files()` command rather than using `add_dependencies()`?

---

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 14, 2021, 11:20am UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/3 "2021-06-14T11:20:02Z")

</div>

Excuse me my poor example. Could you assume “lib2.c might include or might not include generated.h”  
If lib2.c doesn’t include generated.h, the action header\_gen isn’t executed.

In my example, each target has one source file.  
But in practice, each target has bunch of sources and the project has hundreds of targets. The project has thousands of sources.

In such a case, prescanner can discover independent files in lib2 and Ninja dyndep can schedule them in prior to generated\_h.  
Or, generated\_h stays as a bottleneck.

I am proposing the way to give opportunity of dyndep.

---

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 14, 2021, 11:23am UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/4 "2021-06-14T11:23:27Z")

</div>

FYI, I experimented “header-only interface library”.

```auto
# inc/CMakeLists.txt
add_library(generated_h INTERFACE ${generated_file})
target_include_directories(generated_h INTERFACE ${CMAKE_CURRENT_BINARY_DIR})

```

```auto
# CMakeLists.txt
add_library(lib2 lib2.c) # depends on generated header
target_link_libraries(lib2 PRIVATE generated_h)

```

This can prune lib12’s dependency on generated\_h, since lib12 uses target\_link\_libraries(INTERFACE)  
But couldn’t prune main’s dependency.

It’s my intention that neither lib12 nor main depend on generated\_h.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [June 14, 2021, 11:29am UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/5 "2021-06-14T11:29:36Z")

</div>

It sounds like [this issue](https://gitlab.kitware.com/cmake/cmake/-/issues/17097) is of interest here. Also note that it is only really possible with the Ninja generator(s).

---

<div class="post-metadata">

**Author:** ![marc.chevrier](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/m/ecb155/32.png) [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)\
**Post date:** [June 14, 2021, 11:37am UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/6 "2021-06-14T11:37:50Z")

</div>

Ok, I understood your point.

I am afraid this is not the meaning of the story… Until `3.20`, for makefile generators, a custom scanner was used to discover dependencies for c/c++ files, but this approach has many drawbacks:

- This scanner is not able to handle all the complexity of the languages and preprocessor directives: a **lot** of bugs were logged due to this approach.
- Each new language requires to develop a specific scanner which is nearly impossible with a reasonable cost.

So, the future is to rely on compilers themselves to generate implicit dependencies.

---

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 15, 2021, 9:47pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/7 "2021-06-15T21:47:35Z")

</div>

I guess my proposal will resolve it but it is not only an issue I want to resolve.

I took a way to introduce another semantics against add\_dependencies, since I knew issues.  
(This is because I abandoned the way to dissolve and extract dyndep from order-only deps chanin)

- Introduce semantics to local order-only deps, out of add\_dependencies.
  - It may be transitive. It should be available as PRIVATE as least.

- Dyndep-capable generator(s), like ninja, can substitute it to dyndep scanner.

Do you think other generators could handle local order-only deps?

---

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 15, 2021, 9:58pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/8 "2021-06-15T21:58:40Z")

</div>

Also I was afraid of diversion of _cmake\_depends_. I know CMake is getting rid of relying of it.

But I also thought that existent codebase might be considered to divert.

In my proposal,

- If a prescanner is specified, the facility is activated. I think like CMAKE\_EXPERIMENTAL\_LANG\_PRESCANNER  
\*\* Clang will provide compatible scanner, clang-scan-deps, for it in the future.
- _cmake\_depends_ may be enhanced as default scanner, when “awareness of missing files” is implemented.

Prescanner may be “gross scanner”. I think _cmake\_depends_ will be acceptable to it.

---

<div class="post-metadata">

**Author:** ![chapuni](https://discourse.cmake.org/user_avatar/discourse.cmake.org/chapuni/32/1555_2.png) [@chapuni](https://discourse.cmake.org/u/chapuni)\
**Post date:** [June 15, 2021, 10:00pm UTC](https://discourse.cmake.org/t/proposal-dependency-pre-scanner-for-generated-includes/3558/9 "2021-06-15T22:00:35Z")

</div>

FYI, I found a similar issue.

> [@Automatically detecting generated dependencies](https://discourse.cmake.org/t/automatically-detecting-generated-dependencies/1183/7):
>
> I saw a topic reference this one and I thought I’d suggest some functionality for CMake that would be useful. We couldn’t go with CMake largely because of this problem and instead went with an in-house solution. We can declare virtual header files that the dependency scanner is aware of. These files are not guaranteed to exist during scanning and are not scanned themselves; nonetheless, the proper dependency is made between the source file and the various generated headers. It would be nice if C…

My proposal will help him, but seems too late.