# Concurrent running FetchContent\_MakeAvailable() with shared SOURCE\_DIR
**URL:** https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726
**Category:** Usage
**Created:** [May 24, 2022, 8:27pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726 "2022-05-24T20:27:47Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@gerhardol](https://discourse.cmake.org/u/gerhardol)
#### Post date: [May 24, 2022, 8:27pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/1 "2022-05-24T20:27:47Z")
We fetch some large SDKs (several GB), to speedup builds and avoid full disks the SDKs should be put on a shared and persistent path. The code is not too complicated and uses `flock` to make sure that only one version at a time is installed (which could happen on CI servers).
I would like to use `FetchContent_MakeAvailable()` to do this instead, setting SOURCE\_DIR to the persistent path (no binary path right now), but do not see that the command is protected from concurrent installations. It works fine with local paths of course.
Did I miss something or ae there other ways to ensure concurrency?
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [May 24, 2022, 10:18pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/2 "2022-05-24T22:18:23Z")
I’m not clear on what you’re potentially running concurrently. The CMake execution is (currently) inherently single threaded. While I have future plans to potentially make `FetchContent_MakeAvailable()` (or some other equivalent new function) be able to process its dependency list in parallel, that doesn’t apply right now.
If you’re talking about running multiple instances of CMake at once, then that is indeed asking for trouble. If you are sharing things between such multiple instances, you are also responsible for ensuring they don’t interfere with each other.
I’ve seen people use things like [DoIt](https://github.com/pydoit/doit) to parallelise artefact downloads during generation. That’s probably a tangent to your main query here though.
---
### Author:  [@gerhardol](https://discourse.cmake.org/u/gerhardol)
#### Post date: [May 24, 2022, 10:37pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/3 "2022-05-24T22:37:25Z")
Yes, several builds so several CMake processes.
I guess a semaphore could be implemented too, to use CMake instead of flock/curl (but that will increase the complexity).
Slightly outside intended use of CMake, will set the response as a solution.
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [May 24, 2022, 10:43pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/4 "2022-05-24T22:43:17Z")
A change I’m aiming to have included in CMake 3.24 may help you here. If you wanted to do your own sort of locking, the new dependency providers feature would allow you to wrap each call to `FetchContent_MakeAvailable()`. You could do some kind of file locking, for example, but obtaining the lock, forwarding the call back to `FetchContent_MakeAvailable()` again, then releasing the lock. The implementation is smart enough to detect the recursive call and not put you in an infinite loop.
It’s an interesting use case for that feature.
[https://gitlab.kitware.com/cmake/cmake/-/merge\_requests/7276](https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7276)
---
### Author:  [@gerhardol](https://discourse.cmake.org/u/gerhardol)
#### Post date: [May 25, 2022, 8:32am UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/5 "2022-05-25T08:32:17Z")
Thanks!
A quick review of the documentation does not make this completely clear to me right now, will try to look at the change later (even if not in time for the PR).
---
### Author:  [@gerhardol](https://discourse.cmake.org/u/gerhardol)
#### Post date: [October 12, 2022, 11:14am UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/6 "2022-10-12T11:14:25Z")
A follow up if anyone is interested.
A summary is that without some additional implementation, this usecase has really no benefit from using custom scripts.
Implemented similar to the example in the doc, just adding CMake locking
[https://cmake.org/cmake/help/latest/command/cmake\_language.html#id3](https://cmake.org/cmake/help/latest/command/cmake_language.html#id3)
```auto
file(
LOCK ${GOOGLETEST_SOURCE_DIR} DIRECTORY
GUARD FILE
TIMEOUT 600
)
```
(An explicit RELEASE is done too).
I was not sure how to use `file(LOCK)`, there are not much examples, but FILE seem to be working.
To not make unnecessary locks, the lockfile is specific to the dir to installed, and that file need to be added to a variable. (I would have preferred to get the “FetchContents SOURCE\_DIR” to be available, but have not found out how.)
One intention with the change was to be able to test with GoogleTest offline. That does not work as populate will always try to connect, also when the repo also exists with the correct sha. That may be fixed with a special check that sets the proper variables without calling the built-in implementation.
My initial intention with the change was to replace a separate script download for dependencies. That was not possible as some setup is done prior to project() and I do not want to copy that into the product repo, at least right now.
Summary of limitations:
- Get SOURCE\_DIR (and more?)
- How to avoid additional downloads
- Handle “hooks” prior to project() (I expect that to be reject, if ever raised).
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [October 16, 2022, 10:53am UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/7 "2022-10-16T10:53:31Z")
> [@gerhardol](#):
>
> One intention with the change was to be able to test with GoogleTest offline. That does not work as populate will always try to connect, also when the repo also exists with the correct sha.
It shouldn’t need to contact the remote after the first run successfully downloaded it in this scenario, but you’re probably experiencing [this bug](https://gitlab.kitware.com/cmake/cmake/-/issues/21146).
---
### Author:  [@gerhardol](https://discourse.cmake.org/u/gerhardol)
#### Post date: [October 21, 2022, 2:50pm UTC](https://discourse.cmake.org/t/concurrent-running-fetchcontent-makeavailable-with-shared-source-dir/5726/8 "2022-10-21T14:50:57Z")
Thanks again!
If someone else reading this is unsure of the status for the issue, fetch\_contents will delete the directory and redownload it. (Two times in my setup for some reason). This is also with a Git SHA, with 3.24.2.
(We initially used a tag and PATCH\_COMMAND too, but using sha for a committed change is not enough.)
–
Other observations:
Using an URL will not work either (that is more obvious, just had to test).
Download is done twice in our setup. Something triggers configure to run again. “Re-running CMake…”. some custom targets (like running git-describe were running twice too. That is a different problem. (tried to debug ninja setup)
In some situations not just the dir to download to was deleted, but also the parent directory. I cannot reliably recreate. The global download dir should be gtest/sha so it is not a too bad issue in my setup.
So I am keeping the separate download script for now, may look into to this myself as well.