Inconsistent behavior of Ninja generator on first and subsequent cmake runs for CMAKE_<LANG>_STANDARD_LIBRARIES

Code
, ,
1

Hi, we are writing a custom MSVC toolchain (specifically for Visual Studio 2017) and while doing so, we’ve run into a problem where generated build.ninja file is different on first cmake run compared to subsequent runs.

After first run it generates following (stripped down for readability) build statement:

build test.exe: CXX_EXECUTABLE_LINKER__test_Debug CMakeFiles\test.dir\main.cpp.obj
  LINK_LIBRARIES = kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib

However, when build directory is regenerated, LINK_LIBRARIES variable setting disappears. Note, I have manully removed all variables except LINK_LIBRARIES, as they remain the same after regeneration and would just get in the way.

This is generally not a problem, as link.exe will include those libraries in default libraries set by default, so as long as they are found using library search paths, things work and nothing needs to be mentioned on the command line.

We have noticed this because in our toolchain file we set CMAKE_<LANG>_STANDARD_LIBRARIES to contain these default libraries:

set(CMAKE_C_STANDARD_LIBRARIES
    kernel32.lib
    user32.lib
    gdi32.lib
    winspool.lib
    shell32.lib
    ole32.lib
    oleaut32.lib
    uuid.lib
    comdlg32.lib
    advapi32.lib
)
set(CMAKE_CXX_STANDARD_LIBRARIES ${CMAKE_C_STANDARD_LIBRARIES})

What happened was that builds would succeed without any problem after first build directory generation, but then after build directory got regenerated, builds would fail. Looking into it, we found out that regenerating changed the LINK_LIBRARIES in build.ninja file to following:

build test.exe: CXX_EXECUTABLE_LINKER__test_Debug CMakeFiles\test.dir\main.cpp.obj
  LINK_LIBRARIES = kernel32.lib;user32.lib;gdi32.lib;winspool.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;comdlg32.lib;advapi32.lib

That is, instead of separating libraries using spaces, it uses ; (cmake list syntax).

Based on this, it looks like the behavior is to

  • on first generation, ignore CMAKE_<LANG>_STANDARD_LIBRARIES and use hard-coded value
  • on subsequent generation, use CMAKE_<LANG>_STANDARD_LIBRARIES assuming it holds a string, rather than a list

Now the discrepancy between first and subequent run seems like a bug, but it would be nice to hear someone else’ opinion on the matter before reporting it as such.

Treating CMAKE_<LANG>_STANDARD_LIBRARIES as string, rather than list, may be intended, but it seems to be inconsistent with CMAKE_<LANG>_STANDARD_INCLUDE_DIRECTORIES, which is treated as list. Should that also be considered a bug? On the other hand, CMAKE_<TARGET_TYPE>_LINKER_FLAGS_INIT are also treated as string rather than list, but at least the behavior seems to be consistent and first and subsequent cmake runs.

Note, as a workaround, we can set CMAKE_<LANG>_STANDARD_LIBRARIES as string using following:

string(JOIN " " CMAKE_C_STANDARD_LIBRARIES
    kernel32.lib
    user32.lib
    gdi32.lib
    winspool.lib
    shell32.lib
    ole32.lib
    oleaut32.lib
    uuid.lib
    comdlg32.lib
    advapi32.lib
)
set(CMAKE_CXX_STANDARD_LIBRARIES "${CMAKE_C_STANDARD_LIBRARIES}")

This produces the same value of LINK_LIBRARIES in build.ninja for first and also subsequent runs, even though the process of arriving at that value is different.

2

Toolchain files are only read on the first configure (well, whenever a new language is being enabled, really). Since it sets this up via a local variable, on subsequent runs it disappears because the toolchain file is not read and nothing sets the local variable.

3

True, they should only be read on the first configure, but I also saw the value of that variable stored in the CMakeCache.txt which gives access to it in subsequent runs.

But now that you’ve brought it up, I experimented with it a little more. I added a message() call to toolchain file and CMakeLists.txt to both see what the value of CMAKE_<LANG>_STANDARD_LIBRARIES is and to see whether toolchain is indeed only read once. Findings are a bit surprising to me.

First interesting thing is that toolchain file is also read on subsequent configure runs, even if they were triggered due to changes in CMakeLists.txt and not manually from command line. On first run, toolchain file is executed twice, while on subsequent runs it executes only once. I did not expect it to be used in subsequent runs unless I enable more languages.

Second is that I think I know what is happening now. I made a wrong assumption that the way CMAKE_<LANG>_STANDARD_LIBRARIES is meant to be used is to set it in toolchain file and CMake (when it does compiler checks) will use this value and, if everything passes, store it in cache. But that doesn’t seem to be the case and instead the variable follows normal scoping rules, which appear inconsistent due to differences between first and subsequent runs. CMake just seems to ignore it during compiler checks and then stores into cache whatever it deems correct.

On first run, value set by toolchain file is shadowed by the value set in cache by compiler checks, and this cached value is then used to generated build files. However, because cache variable is already set and compiler checks don’t happen on subsequent runs, and because toolchain file is re-read on subsequent runs, setting of the variable in toolchain file will overshadow its value in cache and reach generated build files.

I guess this explains surprising behavior I was seeing and correct approach in this case should be to not set the CMAKE_<LANG>_STANDARD_LIBRARIES variable in toolchain file at all and leave compiler checks to set it to default. Though I still don’t know why is toolchain re-read on subsequent configure runs and whether that’s a mistake on my end or normal behavior.

4

Are you sure about that? This doesn’t sound right. Toolchain files frequently (I’d go so far as to say usually) set non-cache variables. I’m pretty sure they will always be read at least once on every CMake run. It may be read more times on the first run, since it gets used in try_compile() calls when enabling a new language that hasn’t been enabled in that build directory before.

5

I was trying to find some official documentation on when is toolchain file used exactly, but I couldn’t. I think the idea (at least for me) came from

  • toolchain file only needs to be specified on first run
  • toolchain file is invoked on first project() call or when enabling languages that weren’t enabled before

But that doesn’t imply that it is called only when project() is called or language is enabled.

6

My mental model could be wrong… Maybe I’m improperly extrapolating from/conflating with “make sure you first-configure with your toolchain file of choice”…