This is a recurring question but it does not seem to have a satisfying “generic answer”:
At build time, can I detect that new source or header files have been added?
In this case, how can I add them to the project, in a way that subsequent configuration/generation/build/install is limited to a minimum (typically, not rebuilding all translation units that are not dependent on the new files (i.e. they should not have been modified)).
I saw, from a similar question, a link to that: Generated Sources In CMake Builds - Crascit.
It seems to open a path to a possible 100% CMake solution.
In my use case, I’m calling CMake through python script, so it may be acceptable that I add python code in my script to process this detection and any required update in CMake files.
Yet a 100% CMake solution would be more satisfying.
Additional requirement: this check should be as lightweight as possible (otherwise the obvious, expensive, solution, would be to reconfigure at each build and always rebuild everything, which is not acceptable).
NB In my case, I think it can be OK to suppose that every files in given locations are to be added (as would do the GLOB not recommended solution) and that exceptions could be treated manually in the CMakeLists.txt or by putting “not-to-be-added” files in another location.
The behavior you’re looking for should already be what is happening. If you add a new source or header file, presumably that means you’ve had to edit a CMakeLists.txt file somewhere to add it to a target. The next time you do a build, CMake will see that modified CMakeLists.txt file and it will automatically re-run CMake. That will then add the new file(s) to the build. The build tool then should only be building things that are missing or out-of-date. Simply adding a new file to the build should not on its own cause all files to be recompiled.
Perhaps you’re adding a header that all other files directly or indirectly depend on? If you’re using the Ninja generator, here’s a set of steps you can try to get some insight into what’s happening in your last build step:
Add the new source or header file(s) to your CMakeLists.txt file.
Re-run CMake (we do this explicitly on its own so we can avoid it re-running in the next step).
In your build directory run ninja -d explain and it will tell you why it is doing what it is doing.
The output from ninja -d explain can be quite large, but it may give you some clues as to why more things are being rebuilt than you’re expecting.
I would have like, also to avoid to go through an expensive CMake configuration and generation again. For instance there might be a lot of introspection tests that are useless to rerun. But I will leave it alone for now, as it may be a bad caching of some variables on my side…
My main concern is more, when I’ m building, to detect that a new source file has been added and must be recompiled and linked to the final target.
Is it possible to do so with a cmake command? The nuclear weaponry would have to use my python launcher to look at the current sources list in the CMakeLists (possibly in a simpler auxiliary file) and compare it to the current directory content. If there is a difference => update the source list (thus the CMakeLists is modified and the subsequent build command (cmake --build .) should rerun CMake.
Is it a valid way to go, is there a better solution in your opinion?
It feels like you’re fighting the wrong problem. If the list of sources changes at all, CMake has to be re-run. You can’t bypass that (if you do, you’re on your own…). If you are listing your sources explicitly in your CMakeLists.txt files, you don’t have to do anything, CMake will be re-run automatically by the build tool if you do a build after you modify any CMakeLists.txt in the project.
If you are saying you want to be lazy and not have to manually edit the CMakeLists.txt file when a new file is added… you’ve probably already expended far more effort trying to do that than simply expecting people to manually add files in such cases. I’d really recommend you move away from this mind set of trying to avoid manually updating the CMakeLists.txt files when sources are added. The approach you’re taking has all the same drawbacks as using file globbing to obtain your list of sources for a target. It makes assumptions that won’t always hold true, like someone temporarily copying a source file for local editing, and they don’t want that file compiled yet. Or having some sources that only compile if some condition is true (a particular platform or enabled option).
You would normally cache the result of such tests so that subsequent runs don’t have to repeat them.
I’m providing a wrapper above CMake for non expert users. Without going into details, it happened several times that they add new sources into their project and forgot to update their CMakeLists accordingly. Thus my need…
Yet your raised concern about temporary or misplaced file is valid through the actual (bad) solution based on globbing is suffering from the same issue.
A solution would be, when I’m detecting a change in the file list, to prompt the user for confirmation…
Ultimately though: if a developer creates a new source and integrates it, the either compilation or linking would typically fail, right?
Anyway, you might be able to compare the entries from the compilation database with the files you see on disk. If there’s a mismatch, issue warning.
If I would do such a thing, I’d do it as a post build step, to first give the build the opportunity to fail.
You could even make it fancy and only look add (un)staged or recently added files (or files not on your main branch), by running some Git commands. Might make it a bit slower, but potentially saves you from having to make an ignore list.
If I’m building a library, it may not fail to compile until the library is tested (I can declare a new function and not define it, only the client will fail to link with undefined reference).