The use case I am interested in is improving the CMake dependency lookup mechanism in meson (not CMake subprojects but the CMake dependency / find_package() backend). What we are currently doing for using the *-config.cmake and Find*.cmake ecosystem is to generate a minimal CMakeLists.txt with basically only a find_package("${MESON_DEP_NAME}") and some helpers. We then use the CMake trace to extract all the required information for the dependency.
The problem is that CMake invokes the compiler for each dependency multiple times to gather data. This is naturally fine for configuring a real CMake project but slows down the dependency detection in meson considerabely (especially on Windows with MSVC). Additionally, meson already has all the required compiler and system information since meson is also a build system and could pass all this to CMake.
Not providing any languages to CMake also doesn’t work because this breaks some dependencies that rely on the presents of some languages and use try_compile().
The previous solution/hack was to create a fake, already “configured” CMake build directory. However, this code path is really fragile (for obvious reasons) and may break with future CMake versions. We have also switched to generating/using CMake toolchain files to also provide better cross compilation support. Unfortunately this switch has lead to some performance regressions: https://github.com/mesonbuild/meson/issues/7898
The real fix would be to have some API in CMake itself for providing the same information in the future, however, we still also want to support older CMake versions for compatebility reasons.
I don’t know about that because CMake extracts a lot of information from compilers that I’d be surprised if Meson cared much about. Please look at the CMakeFiles/$version/{CMakeSystem,CMake${lang}Compiler}.cmake files for things that CMake cares about. I don’t know how much of this can be stuffed into the cache ahead of time, but I doubt meson is going to be able to list out the _COMPILE_FEATURE variables well enough to be accurate. You could probably pre-populate some of it, but there’s no guarantee that future versions of CMake won’t want to have additional information either.
I don’t know what happens if you provide partial information, but fiddling around too much here is liable to get into sensitive areas where our compatibility guarantee does not hold so well.
Also, how are you extracting information? With imported targets being the norm now, there are issues with extracting information reliably. See this issue. In ParaView, we have some consumers that don’t use CMake and I ended up making this script to extract the information. It basically makes a full project and then asks the resulting build what’s going on and computes a diff of command line flags for a target which uses the targets in question with those that do not. Not pretty, but it’s the best that can be done with the tools CMake offers today.
We are generating a CMake project (see our CMakeLists.txt for that) which we then configure with all the necessary input variables set. Everything past line 25 is for the old style, non-target dependencies to ensure that we track the required variables correctly (we basically only track the set() function for variables state and some packages use string() to set some variables, hence we work around this by copying them to new variables)
We generally only rely on the CMake --trace --trace-expand (and --trace-format=json-v1 if supported) to extract the dependency information. We then parse the trace by reimplementing the subset of CMake functions that are required to keep track of imported targets and properties. See traceparser.py#L105 for a complete list of functions we implement.
These functions are then “executed” in the order they are in the trace output. This way we reconstruct a partial CMake state with all variables (assuming they are set with set() and not some fancy string() or list() function) and all imported targets (we can even keep track of the dependency tree in INTERFACE_LINK_LIBRARIES). Meson then uses this state to generate the compile and link flags.
This system works supprisingly well. We have never gotten a bug report because of incomplete/wrong extracted information, just issues about actually finding the dependencies by calling CMake correctly (CMAKE_MODULE_PATH, and other fun stuff).
What I basically want is a flag to tell CMake “I know what I am doing, yes this can break in a real project, but I really don’t want you to check any compilers because it is slow and 90% of the information you get is irrelevant for dependency lookups anyway”
The best solution would of course be a resolution to #17236 that would provide all the information we currently get from our traceparser.
We also have a minimal parser for generator expression: generator.py
It is by no means complete (or even guaranteed to be correct) but it works for most use cases we need to support and is only updated when we need new functionality.
That seems like a more reasonable approach to the “what does CMake think is going on”. If you want to skip a lot of compiler checks, I’d say that in addition to passing basic information via -D, copying the CMakeFiles/$version directory over ahead of time from an already-working build tree may save you a lot of trouble. It would also be backwards compatible. I don’t know if CMake completely ignores it for new trees, so you might want to write a simple CMakeCache.txt directly instead of passing -D flags for compiler information.
If you want a more demanding test case, I’d be interested to see how things work for find_package(VTK) to extract information about how to use, say, VTK::CommonCore, VTK::CommonDataModel, and VTK::RenderingOpenGL2.
That is more or less what we were doing in previous versions, but we were actually generating these files ourselves instead of using a “real” CMake build directory. This used to work more or less OK, but felt like a really bad hack.
I think I’d use a real directory. That gets rid of the guesswork and (most of?) the incompatibility question.
Ah, I should have been more specific: VTK 9.0 or newer. The build system was rewritten to make use of CMake’s targets (rather than home-grown variable management to simulate them). Granted, the system was used as a guide for what targets could be, but VTK needed to jump to make use of what they became.
I guess I can do it locally as well now that I have a template .
Thanks. The log file is in <build dir>/meson-logs/meson-log.txt. Meson 0.55.x still uses the fake CMake build dir. The new toolchain file version is new in 0.56.x (rc1 and rc2 are on pypi)
I guess this could work. The reason why we are not doing this currently is because CMake likes to cache stuff, especially some dependency variables. I don’t know/want to find out if this can cause issues when changing the project name and detecting multiple dependencies.
We are calling CMake once anyway for extracting system information. I could try using this build dir and see if there are any issues in our tests. See our CMakePathInfo.txt.
It looks like it works pretty well for VTK . It does warn about dl not being known, but I guess that’s a problem with that not having a target for us to use for that.
I’d cache the compiler extracting information and just pass the compilers via -D. I think I’d try and transfer as little of the CMakeCache.txt as possible.
I also noticed that on my side. Is dl a special target in CMake or are all non-tragets just appended with an -l prefix?
I could try that, thanks.
Also are there any plans on the CMake side to have better pkg-config / pkgconf support? While I did manage to get the dependency information from CMake, our aproach is still very far from official and convinient.
Alternatively, if *.pc files are to limiting, would be a custom, new format (that doesn’t require a full CMake parser + moddeling the internal CMake state) be an option?
Any non-path, non-target, non-flag item is assumed to be in some standard location, so CMake just prefixes -l (or leaves it as is on Windows like with opengl32.lib) for those items and lets the linker figure it out.
There have been discussions about that, but Matthew doesn’t seem to be on Discourse? The main issue tracker is here. I don’t know if there’s already a Discourse thread or not.
.