We recently started to move some of the manual dependent-repo fetching we were doing over to be automatically pulled in via FetchContent. It’s definitely a lot easier when everything’s automatic, but the initial fetching (and after any time someone does a clean build since the fetched sources are stored in the build tree) is much slower than it used to be just fetching them normally since cmake configure is fetching them one by one and not in parallel.
Is there any way to make this happen in parallel? It would speed up the fetching of these external dependencies a lot.
There’s a couple of aspects involved here. Firstly, the serial performance of FetchContent population is slow on some platforms (Windows especially, potentially macOS as well if the Xcode generator is used). I had a fix for that, but had to revert it due to some backward compatibility problems. There’s a way forward for that now, I just haven’t had the time to get back to it and put the changes together.
The second part, which is more aligned with your question, is that CMake currently doesn’t have a way to do things in parallel during the configuration step. I’ve had some initial private discussions about a way to give dependency providers the ability to supply a set of dependencies in one go, which would allow them to do their work in parallel. My current goal is to put up a proposal for that feature in the next few months. There are parts of that work which may have follow-on benefits for FetchContent itself being able to do some things in parallel, but I need more time to think through that aspect.
Is there any way to make FetchContent not do a build, and to literally just download instead? I ran cmake in profiling mode and opened the trace in chrome://tracing, and I see something like this:
Then, looking at the specific command line it’s executing:
I think the performance problems would be solved it weren’t trying to actually build the subproject. The user is just running configure here, not build. So why does the subproject need to be built during configure?
I looked at the source for FetchContent and tried to step backwards through the code at the lines mentioned in the callstack and I can’t find any obvious way to make it skip this step. Our intention is literally just to clone a repo and then add_subdirectory() it, so if you have any other suggestions on how we might accomplish that that doesn’t involve FetchContent, that might solve our problems too.
While it may appear that this is doing a build, you are misunderstanding how FetchContent works. At the moment, it does the download, update and patch steps by creating a sub-build which uses ExternalProject to do those things. What you’ve identified as a build is in fact just the implementation of the download, update and patch. My previous comment about fixing the serial performance of FetchContent directly relates to that. The improvement that had to be reverted but which now has a way forward was in fact bypassing that separate sub-build and doing the necessary steps directly in the main CMake configure process. You won’t see much difference on Linux, but you will on Windows and likely will if you are using Xcode. No ETA on when I’ll get a chance to implement that though.
TLDR: It is actually doing what you want, it’s just that the implementation currently used is inefficient, especially so on Windows or if using the Xcode generator.
Hi @craig.scott, any updates on the effort of speeding up fetch content?
I’ve have several precompiled artifacts and I use fetch content to download/decompress them, I need those at configuration step due to some target dependencies.
I could try to use a different approach to fetch the content myself, but found the current functionality actually nice and would be a shame not to use it or deviate from it.
