# Good way of running custom commands serially

**URL:** <https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412>\
**Category:** Code\
**Created:** [June 27, 2023, 1:25am UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412 "2023-06-27T01:25:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![AnthonyD973](https://discourse.cmake.org/user_avatar/discourse.cmake.org/anthonyd973/32/3130_2.png) [@AnthonyD973](https://discourse.cmake.org/u/AnthonyD973)\
**Post date:** [June 27, 2023, 1:25am UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/1 "2023-06-27T01:25:20Z")

</div>

**TL;DR** : I `add_custom_command`s that call `git clang-format`. They are called as part of target (e.g. library) build. The commands don’t create/modify any files but can’t run concurrently. How to enforce only one to run at a time?

* * *

I’m writing a CMake module which runs `git clang-format` on the sources of user-specified targets. One way that `git clang-format` is used is during the build of the target (it only checks the formatting; doesn’t create/modify anything).

The problem is that `git`, for very valid reasons, does not like to have multiple processes running on the same repo, and therefore errors out. So I need to run each command serially.

I thought about creating an `add_custom_target()` for each custom command, and making a chain of `DEPENDS` between them. But that would not work: not all custom commands should always be run, since the user-specified targets don’t always build (e.g. if lib1’s sources change but not lib2’s, only the checks of lib1 should run).

I also noticed the `JOB_POOL` on `add_custom_command()`, but that is Ninja-only, whereas obviously I’d want the CMake module to work with multiple generators.

I’ve seen [this thread](https://discourse.cmake.org/t/serial-custom-commands/7821), but the solution would not apply to this scenario.

So what good way is there of doing this? Or should I just run `git clang-format` differently?

FWIW, [this the CMake module in question](https://github.com/adentinger/AdeClangFormat).

---

<div class="post-metadata">

**Author:** ![jtxa](https://discourse.cmake.org/user_avatar/discourse.cmake.org/jtxa/32/1535_2.png) [@jtxa](https://discourse.cmake.org/u/jtxa)\
**Post date:** [June 29, 2023, 11:28pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/2 "2023-06-29T23:28:58Z")

</div>

CMake generates build files, the commands are executed by the build tool. If that tool is incapable of avoiding parallelization, I guess there’s nothing CMake can do.  
Either the user has to call the build without parallelization, or you need to wrap your command with some semaphore. Like `flock` on Linux.

But the other question is, why can’t you execute it in parallel. For read only actions the commands should not fail, otherwise I would expect some highlighted warning in the documentation.

---

<div class="post-metadata">

**Author:** ![AnthonyD973](https://discourse.cmake.org/user_avatar/discourse.cmake.org/anthonyd973/32/3130_2.png) [@AnthonyD973](https://discourse.cmake.org/u/AnthonyD973)\
**Post date:** [June 30, 2023, 11:34am UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/3 "2023-06-30T11:34:54Z")

</div>

Yeah, running the command that way seems like an appropriate solution. I’ll give it a try; thanks!

As for why the commands can’t be run in parallel given they are read-only, I’m not sure, but since `git clang-format` is an extenstion of Git’s CLI (a python script called `git-clang-format`), it’s possible that the script doesn’t indicate that it is read-only, or that Git just doesn’t support having multiple processes running on the same repo.

---

<div class="post-metadata">

**Author:** ![fdk17](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/f/ea666f/32.png) [@fdk17](https://discourse.cmake.org/u/fdk17)\
**Post date:** [July 1, 2023, 8:25pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/4 "2023-07-01T20:25:55Z")

</div>

When I use add\_custom\_command which multiple commands they all run serially as described in the CMake manual. Or you could have a script that runs all the desired commands which is also described in the add\_custom\_command section.

---

<div class="post-metadata">

**Author:** ![jtxa](https://discourse.cmake.org/user_avatar/discourse.cmake.org/jtxa/32/1535_2.png) [@jtxa](https://discourse.cmake.org/u/jtxa)\
**Post date:** [July 1, 2023, 8:42pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/5 "2023-07-01T20:42:07Z")

</div>

That has the same drawback as adding dependencies. I guess you missed that sentence:

> [@AnthonyD973](#):
>
> I thought about creating an `add_custom_target()` for each custom command, and making a chain of `DEPENDS` between them. But that would not work: not all custom commands should always be run, since the user-specified targets don’t always build (e.g. if lib1’s sources change but not lib2’s, only the checks of lib1 should run).

The execution shall be independent from each other, but if multiple of them are called, they shall not be in parallel. `JOB_POOL` would be the right solution, but it just works for Ninja.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [July 1, 2023, 8:56pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/6 "2023-07-01T20:56:54Z")

</div>

Job pools are the only reliable mechanism here, but that is `ninja`-only.

However, I would caution against such a target being on in non-developer builds as `clang-format` is not stable over time and someone with `clang-format-6` may have problems if `clang-format-10` has made a pass over it. These kinds of things are (IMO) best done in CI or some other central place prior to merging updates rather than as part of the build or as a commit hook.

---

<div class="post-metadata">

**Author:** ![AnthonyD973](https://discourse.cmake.org/user_avatar/discourse.cmake.org/anthonyd973/32/3130_2.png) [@AnthonyD973](https://discourse.cmake.org/u/AnthonyD973)\
**Post date:** [July 1, 2023, 9:44pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/7 "2023-07-01T21:44:44Z")

</div>

Thanks for the help! From all the replies, I think running the command as part of a file-locking mechanism (like `flock`) would be the only viable solution. Platform compatibility is likely to be an issue (not sure if a readily-available equivalent would exist on Windows), but at least there _is_ a solution.

> [@ben.boeckel](#):
>
> However, I would caution against such a target being on in non-developer builds as `clang-format` is not stable over time

Letting myself get sidetracked here: yes, for example, I’ve seen [this post](https://discourse.cmake.org/t/clang-format-integration/3358/6) you’ve made about this specific point. In fact, [that’s exactly why my CMake module adds support for `git-clang-format`](https://github.com/adentinger/AdeClangFormat/blob/f04b87a8b9f33ff0e3436e45d676a5de4f55c7c8/doc/git-clang-format.md), rather than only plain `clang-format`. I plan to use to introduce this to an existing codebase that isn’t formatted properly, and using normal `clang-format` would require destroying version control history, even if we put the format checks only on the CI. Although, when using `git-clang-format` in production, I’m not sure whether automatic fixes might introduce bugs in the code or not, with the whole “consider only the modified lines” thing. I would guess not, but `git-clang-format` doesn’t seem too widely used so I wouldn’t trust it that easily.

This issue here is that, unlike normal `clang-format`, `git-clang-format` relies on `git`, which breaks when multiple processes are run simultaneously.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [July 1, 2023, 9:48pm UTC](https://discourse.cmake.org/t/good-way-of-running-custom-commands-serially/8412/8 "2023-07-01T21:48:45Z")

</div>

> [@AnthonyD973](#):
>
> and using normal `clang-format` would require destroying version control history

We have new tools at our disposal:

- [`git blame --ignore-revs-file=`](https://www.git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revs-fileltfilegt)
- `git config blame.ignoreRevsFile` to make it more permanent

Then you can `--author="clang format <nobody@blameless>"` the commit that does the mass reformatting.

Personally, I find mass reformatting and enforcement better than “fixed a typo (and reformatted the file)” style diffs over a longer period of time.