Linking libraries computed at build time

Development
1

I have an interesting use case involving static libraries. Let’s say a particular target needs to link to a set of static libraries that isn’t known when CMake is run. The project builds a tool which scans files and works out a set of static plugin libraries that need to be linked to the target (only needed for static builds because the target can load the plugins dynamically at runtime otherwise). The problem is that CMake wants to know the set of libraries at configure time, but the tool that works that out is built later.

A potential solution to this is to tell CMake to add a response file to the linker command line and define a custom command that invokes the tool to write out that response file. The dependencies can be defined to ensure things happen at the right time and are re-invoked when necessary. I’ve got a proof-of-concept working which demonstrates the principle, at least on macOS, but as far as I can tell, it would work with all mainstream linkers.

The questions I have around this technique are the following:

  • CMake currently appears to define an undocumented (so presumably internal) variable CMAKE_<LANG>_RESPONSE_FILE_LINK_FLAG. Would there be any reason we wouldn’t want to document this and support it going forward? If the variable is empty or not defined, that means the linker doesn’t support response files (seems that most linkers do, except for maybe the TI or the gcc 3.x front end, neither of which I think would be a deal-breaker for my use case).
  • Can anyone poke holes in why this wouldn’t work reliably?
  • Are there any other robust ways to link to a set of libraries that are computed at build time?

My use case spans a lot of platforms, including embedded, so it may be challenging to get something that works everywhere.

I don’t know if it would make sense for CMake to offer some kind of direct support for this, but it’s a scenario I’ve encountered more than once. Sometimes these scenarios also involve a set of source files that isn’t known until build time too, but for this particular query, I think I can avoid that. In the past, I’ve used a dedicated sub-build that runs at configure time to get out of jail for this, but in my current scenario, that wouldn’t be practical. It is also a technique that should be a last resort, as it is messy and inefficient.

@brad.king @ben.boeckel

2

The sub-build approach is the only one that can work for VS and Xcode IDE projects AFAIK. The proposed response-file approach can work only for generators where CMake directly generates the link line.

The CMAKE_<LANG>_RESPONSE_FILE_LINK_FLAG variable could be documented for public use for projects to read it, but probably not to write to the variable. Also IIRC it is not always set: generators fall back to @ or some other default if it is not set. Other variables are used to tell the generator whether to use response files for specific contexts.

I’m not sure we should offer direct support for this, but certainly it would be okay to offer the needed building blocks if possible.

3

So after experimenting today, I found that response files work for Xcode, but they don’t for Visual Studio. It may have been possible to create a single source file and embed #pragma comment(lib,...) lines in it as an alternative (also supported by clang apparently, but not gcc). But I couldn’t find one approach that works across all platforms. At best, I could cover the set of platforms with a couple of methods.

Another bigger problem that emerged though is that when taking this route, CMake cannot easily be made aware of the necessary dependencies. The target for which the response file is being used won’t have any dependencies on these extra libraries being added via the response file. In my use case, that is a deal-breaker since the only way to avoid these dependency problems is to make too much of the build depend on things that they don’t otherwise actually depend on. That for me has been the bigger take-away from this exercise.

I’m exploring other ways of solving this particular problem instead of the proposed route, so the motivation for exposing CMAKE_<LANG>_RESPONSE_FILE_LINK_FLAG is no longer there for me now.

4

This can get you .lib files, but I don’t think you can add library paths this way. Or are full paths supported in this form?

Can’t the custom command which generates the file also make a depfile for tracking after the work to enable that? Though that doesn’t work in VS or Xcode I believe…