# How to avoid parallel-build race conditions?

**URL:** <https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727>\
**Category:** Code\
**Created:** [February 27, 2020, 8:32pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727 "2020-02-27T20:32:37Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [February 27, 2020, 8:32pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/1 "2020-02-27T20:32:37Z")

</div>

The documentation of add\_custom\_command states (in part) the following:

“Do not list the output in more than one independent target that may  
build in parallel or the two instances of the rule may conflict  
(instead use the `add_custom_target()` command to drive the command  
and make the other targets depend on that one).”

I have a use case (building some LaTeX-based documentation where both  
intermediate and final products depend on many different generated  
files and associated targets) where I have many interdependent file  
dependencies to insure all custom commands get rebuilt as necessary.  
And the targets have many different target dependencies between them.  
So this complex build system definitely but necessarily has different  
targets depending indirectly on the OUTPUT from a given custom  
command. So to avoid parallel build conflicts (or race conditions)  
for this complex case my interpretation of the parenthesized part of  
the quote above is I have to create custom targets corresponding to  
custom comands and use add\_dependencies between those targets as for  
the simple example below. But could some CMake expert here look at  
this simple example and confirm there are no parallel-build race  
conditions for it?

```auto
# First custom command and associated target
add_custom_command(
OUTPUT
file_a
COMMAND ...
)
add_custom_target(target_a
DEPENDS file_a
)

# Second custom command and associated target
add_custom_command(
OUTPUT
file_b
COMMAND ...
DEPENDS
# This command should be rebuilt whenever file_a is updated
file_a
)
add_custom_target(target_b
DEPENDS file_b
)
add_dependencies(target_b target_a)

# Third custom command and associated target
add_custom_command(
OUTPUT
file_c
COMMAND ...
DEPENDS
# This command should be rebuilt whenever file_a or file_b are updated
file_a
file_b
)
add_custom_target(target_c
DEPENDS file_c
)
add_dependencies(target_c target_b target_a)

```

Given those custom command file DEPENDS are necessary (to get those  
custom commands rebuilt as needed) and those targets are necessary  
(due to other needs for the build system) are the above  
add\_dependencies commands sufficient to insure there are no  
parallel-build race conditions in the above simple example?

Let’s assume the above general pattern with add\_dependencies is  
correct, but I have inadvertently not followed this pattern for my  
complex build system. In this case, is there an unambiguous way with  
CMake of detecting parallel-build race conditions for any given build  
system? Or do build-system implementers that use add\_custom\_command  
have to be constantly on guard about this issue?

Alan

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [February 28, 2020, 8:55pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/2 "2020-02-28T20:55:44Z")

</div>

> [@airwin](#):
>
> In this case, is there an unambiguous way with  
> CMake of detecting parallel-build race conditions for any given build  
> system?

I know of no way to always detect such cases.

With LaTeX in particular, I usually have a chain of `add_custom_command` calls where each chooses an “arbitrary” output file of the command to trigger the next invokation (there is an optimal order, but I suspect it depends on your actual toolchain). I usually end up with 2 `pdflatex`, a `bibtex`, then a final `pdflatex` custom command, the last one having the `.pdf` as its output. That is then hooked up to the custom target. Yes, it’s a mess.

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [February 28, 2020, 10:31pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/3 "2020-02-28T22:31:12Z")

</div>

That’s painful for CMake-based build system developers such as myself  
since it means we have to always be vigilant about the slightest  
change in file dependencies with no proof that we got it right.

I request that the CMake developers address this issue with a CMake  
command-line option that lists all targets available for a given  
configuration, their direct and indirect list of file dependencies,  
and also the subset of those targets where those file dependencies  
clash without add\_dependencies solving that issue.

Let me know if you think this idea is a reasonable one, and  
if so I will write it up in the CMake bug tracker.

Alan

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [February 28, 2020, 10:43pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/4 "2020-02-28T22:43:15Z")

</div>

`add_custom_command` calls where each chooses an “arbitrary” output  
file of the command to trigger the next invokation (there is an  
optimal order, but I suspect it depends on your actual toolchain). I  
usually end up with 2 `pdflatex`, a `bibtex`, then a final  
`pdflatex` custom command, the last one having the `.pdf` as its  
output. That is then hooked up to the custom target. Yes, it’s a  
mess.

For your information I gave up on CMake logic to handle this necessary  
\*tex tool iteration. Instead, I wrote a bash script  
[https://sourceforge.net/p/freeeos/freeeos/ci/master/tree/www/iterate\_pdflatex.sh](https://sourceforge.net/p/freeeos/freeeos/ci/master/tree/www/iterate_pdflatex.sh)  
to perform the pdflatex iteration which anyone is welcome to use (or  
adapt under the terms of the GPL license for FreeEOS). The iteration  
criterion is a general one so I think it should be straightforward to  
generalize this script for all \*tex tools. But I only use pdflatex  
these days so I did not bother with that generalization.

Alan

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [February 29, 2020, 12:17pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/5 "2020-02-29T12:17:27Z")

</div>

Doing this implies that we have accurate _and_ complete dependencies for all commands in the build system. This can only be known post-build in the general case if _everything_ has accurate `depfile` outputs and isn’t “lying” and specify _all_ of their outputs. I don’t think it is feasible for CMake to provide such a tool. The ninja tool is the closest to having some of this information (the deps log), but that is only as good as the tools you’re using report themselves. You need tracking build systems like Tup to do that kind of stuff, but usually then you _have_ to be right and you lose the ability to do things like extract a tarball, use git, etc. during your main build.

---

<div class="post-metadata">

**Author:** ![fdk17](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/f/ea666f/32.png) [@fdk17](https://discourse.cmake.org/u/fdk17)\
**Post date:** [March 1, 2020, 2:56pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/6 "2020-03-01T14:56:26Z")

</div>

There are a lot of may’s in the documentation you are quoting. My experience with custom commands and where the output of the commands are needed by multiple targets is how different generators express this in the project files they generate. I recall seeing that these commands are duplicated for some generators and for others they show up once. This is one of the reasons for the “may” have race conditions. It was my understanding this is why the documentation states to avoid this situation by using a custom target that essentially runs first and then all targets that rely on these outputs run afterwords.

I really think that your example is overly complicated or I don’t understand what you are trying to get at. I don’t see what target\_b or target\_a is for. The build tools like Ninja or Make or Visual Studio wouldn’t have parallelized running the individual commands in my experience because you clearly define the dependencies between them.

For me if I had other targets that depending upon not just on file\_c but on file\_a, file\_b, and file\_c, then I would just use add\_dependencies to have all these other targets be dependent on target\_c.

But the way I read your use-case is that it’s the opposite. It’s sounds like you would just want to build the LaTeX-based documentation last in the entire build process to ensure that all the targets that feed inputs into the documentation target are finished. Then the custom commands for the documentation target will run as necessary if any of their individual inputs were updated.

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [March 1, 2020, 8:27pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/7 "2020-03-01T20:27:38Z")

</div>

@ben.boeckel  
Actually, I don’t believe having complete and accurate depfile outputs  
should be required. In fact, the whole point is to test the custom  
command outputs (however inaccurately and incompletely they may be  
specified) for the current build system!

Also, could you please explain why you think such tests could only be  
done post build?

On the contrary it appears to me that for a given build-system  
configuration, CMake _must_ know every depfile and the associated  
chain of custom commands specified by the CMake-based build system for  
every target. (Or otherwise, it would not know how to configure the  
build of those targets.) So shouldn’t it be straightforward to  
implement an option for CMake that warns whenever two targets that  
have no target dependencies between them refer to the same custom  
command in their two different chains of custom commands?

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [March 1, 2020, 8:56pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/8 "2020-03-01T20:56:21Z")

</div>

@fdk17 said:  
I really think that your example is overly complicated or I don’t  
understand what you are trying to get at.

My use case actually concerns building figures and files for 5  
different LaTeX documents with many common files between the figures  
and the documents. So for my simple example I wanted at least three  
different files with dependencies between them. And in this example,  
I tried to follow exactly the advice in the documentation

```auto
"(instead use the add_custom_target() command to drive the command and
make the other targets depend on that one).”

```

concerning the targets to be implemented for those 3 generated files and the  
target dependencies between those targets.

@fdk17 said:  
The build tools like Ninja or Make or Visual Studio wouldn’t have  
parallelized running the individual commands in my experience because  
you clearly define the dependencies between them.

Thanks for that confirmation that you are pretty sure that following  
this simple rule (if there are common custom commands in the custom  
command chains of two different targets, then serialize those two  
targets with the appropriate add\_dependencies commands) will avoid  
race conditions for my (complex) use case.

Also note that in my response to @ben.boeckel I was only asking that  
an option be implemented for CMake that identifies for a given  
build-system configuration whenever this simple rule is not followed.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [March 2, 2020, 3:51pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/9 "2020-03-02T15:51:01Z")

</div>

Ah. I think I see now. @brad.king Thoughts on feasibility?

---

<div class="post-metadata">

**Author:** ![brad.king](https://discourse.cmake.org/user_avatar/discourse.cmake.org/brad.king/32/11_2.png) [@brad.king](https://discourse.cmake.org/u/brad.king)\
**Post date:** [March 2, 2020, 4:07pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/10 "2020-03-02T16:07:32Z")

</div>

It may be possible to implement a check to detect such races, but we don’t always know they are wrong either. For example a single custom command may be included in two independent targets that are `EXCLUDE_FROM_ALL` such that the project intends only one will ever by built at a time as a helper utility or something.

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [March 2, 2020, 6:24pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/11 "2020-03-02T18:24:15Z")

</div>

@brad.king said:

“It may be possible to implement a check to detect such races, but we  
don’t always know they are wrong either. For example a single custom  
command may be included in two independent targets that are  
`EXCLUDE_FROM_ALL` such that the project intends only one will ever by  
built at a time as a helper utility or something.”

Races are always wrong in my particular use cases (not just for the  
particular use case I mentioned, but many others spread across quite a  
few different projects). The vast majority of my custom targets (all  
configured without the ALL signature) require lots of cpu time to  
build and are typically part of a hierarchy of add\_dependencies on  
some overall target (such as the “FreeEOS\_papers” target for the present  
use case). And I have a computer with 16 hardware threads. So as a  
matter of course I always build such overall custom targets with  
parallel builds. Therefore, I would welcome the capability of  
detecting races for _all_ my use cases for add\_custom\_target, and I  
suspect any other build-system developers that uses add\_custom\_target  
as extensively as I do would also welcome this capability.

@brad.king  
What is the next step that you recommend? Do you plan first to look  
more deeply at feasibility in case this requested capability is  
trivial to implement or do you know already it is going to be hard to  
implement so you would recommend the next step should be for me to  
move this discussion to the bug tracker? Also, if/when that move  
occurs, how should I reference in the bug tracker the present  
preliminary discussion?

---

<div class="post-metadata">

**Author:** ![brad.king](https://discourse.cmake.org/user_avatar/discourse.cmake.org/brad.king/32/11_2.png) [@brad.king](https://discourse.cmake.org/u/brad.king)\
**Post date:** [March 2, 2020, 6:58pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/12 "2020-03-02T18:58:29Z")

</div>

This has been an issue since CMake started. There are probably open issues about it in the issue tracker already. No one has taken the time to try to implement the detection. If someone does I’d be happy to review the work. I do not plan to work on it myself.

---

<div class="post-metadata">

**Author:** ![airwin](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/a/f4b2a3/32.png) [@airwin](https://discourse.cmake.org/u/airwin)\
**Post date:** [March 2, 2020, 8:10pm UTC](https://discourse.cmake.org/t/how-to-avoid-parallel-build-race-conditions/727/13 "2020-03-02T20:10:20Z")

</div>

I have made the feature request at [https://gitlab.kitware.com/cmake/cmake/issues/20412](https://gitlab.kitware.com/cmake/cmake/issues/20412)