Concurrent running FetchContent_MakeAvailable() with shared SOURCE_DIR

We fetch some large SDKs (several GB), to speedup builds and avoid full disks the SDKs should be put on a shared and persistent path. The code is not too complicated and uses flock to make sure that only one version at a time is installed (which could happen on CI servers).

I would like to use FetchContent_MakeAvailable() to do this instead, setting SOURCE_DIR to the persistent path (no binary path right now), but do not see that the command is protected from concurrent installations. It works fine with local paths of course.

Did I miss something or ae there other ways to ensure concurrency?

I’m not clear on what you’re potentially running concurrently. The CMake execution is (currently) inherently single threaded. While I have future plans to potentially make FetchContent_MakeAvailable() (or some other equivalent new function) be able to process its dependency list in parallel, that doesn’t apply right now.

If you’re talking about running multiple instances of CMake at once, then that is indeed asking for trouble. If you are sharing things between such multiple instances, you are also responsible for ensuring they don’t interfere with each other.

I’ve seen people use things like DoIt to parallelise artefact downloads during generation. That’s probably a tangent to your main query here though.

Yes, several builds so several CMake processes.
I guess a semaphore could be implemented too, to use CMake instead of flock/curl (but that will increase the complexity).

Slightly outside intended use of CMake, will set the response as a solution.

A change I’m aiming to have included in CMake 3.24 may help you here. If you wanted to do your own sort of locking, the new dependency providers feature would allow you to wrap each call to FetchContent_MakeAvailable(). You could do some kind of file locking, for example, but obtaining the lock, forwarding the call back to FetchContent_MakeAvailable() again, then releasing the lock. The implementation is smart enough to detect the recursive call and not put you in an infinite loop.

It’s an interesting use case for that feature.

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7276

Thanks!
A quick review of the documentation does not make this completely clear to me right now, will try to look at the change later (even if not in time for the PR).