Do generators expression work with BASE_DIRS in target_sources

I can post a longer example of this issue if needed but I wanted to see if what I am doing is just not expected to work. I have a CMakeLists.txt like

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers 
   TYPE HEADERS
   BASE_DIRS 
        $<BUILD_INTERFACE:${CMAKE_BINARY_DIR}/HelloWorld> 
        $<INSTALL_INTERFACE:${CMAKE_BINARY_DIR}>
   FILES ${CMAKE_BINARY_DIR}/HelloWorld/helloworld_export.h)

When I use this it seems whatever I do the BUILD_INTERFACE is used in the install step. That is the install happens in the root install include directory never in a subdirectory called “HelloWorld” which is what I would have expected.

I assume I am doing something very dumb but in my defense BASE_DIRS is barley documented at all.

So should I expect generator expressions to work in BASE_DIRS?

I don’t think it makes sense to have separate build/install interfaces for BASE_DIRS. What is the goal here? To require helloworld_export.h when including in the source tree, but HelloWorld/helloworld_export.h when it is installed? Given that the header is meant to be included by public headers, how are both spellings going to be satisfied by the files that include it?

Some background. I was upgrading a project to use FILE_SET hopefully to simplify things.
I have attached an example project. I build into an out of source build directory and generate the Visual Studio solution using cmake -G "Visual Studio 17 2022" ../src
Note I can make all this work using some target_include_directories and the like but I was really trying to use FILE_SET as much as possible to learn stuff about it. This might just be pushing FILE_SET usage beyond what it is intended for?

So there is one project HelloWorld that lives in a sub directory HelloWorld The project uses

target_sources(HelloWorld PUBLIC 
   FILE_SET HEADERS 
   BASE_DIRS 
       ${CMAKE_SOURCE_DIR} 
   FILES HelloWorld.h)

for it’s one header. This all works well.
On install the HelloWorld.h header if install in include/HelloWorld/HelloWorld.h which is what I expect. This is using

install(
    TARGETS HelloWorld
    EXPORT HelloWorldExport
    FILE_SET HEADERS
)

The HelloWorld CMake also generates an export header using

generate_export_header(HelloWorld)

The helloworld_export.h is generated in the ${CMAKE_CURRENT_BINARY_DIR} as expected.
Now in HelloWorld.cpp this header is included as

#include "helloworld_export.h"

Using the double quotes in the #include here is the way I have seen in every project I have worked on. So to find this file I need ${CMAKE_CURRENT_BINARY_DIR} in the include path so I can do the following to achieve this

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers 
   TYPE HEADERS
   ${CMAKE_CURRENT_BINARY_DIR}
   FILES ${CMAKE_CURRENT_BINARY_DIR}/helloworld_export.h)

So I add this file set to the install to get

install(
    TARGETS HelloWorld
    EXPORT HelloWorldExport
    FILE_SET HEADERS
    FILE_SET generated_export_headers
)

However the install is then

1>-- Up-to-date: .../install/include/HelloWorld/HelloWorld.h
1>-- Up-to-date: .../install/include/helloworld_export.h

However this is not what I want. I want the helloworld_export.h to be in the HelloWorld directory beside the HelloWorld.h file. So the install path I want includes the HelloWorld directory. This can be done using the following

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers 
   TYPE HEADERS
   BASE_DIRS ${CMAKE_BINARY_DIR}
   FILES ${CMAKE_CURRENT_BINARY_DIR}/helloworld_export.h)

This does what I want on install

1>-- Up-to-date: .../install/include/HelloWorld/HelloWorld.h
1>-- Up-to-date: .../install/include/HelloWorld/helloworld_export.h

However that makes the build fail as it can not longer find helloworld_export.h. So I thought I would use a different BASE_DIR for build and install like so

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers 
   TYPE HEADERS
   BASE_DIRS $<BUILD_LOCAL_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}> $<INSTALL_INTERFACE:${CMAKE_BINARY_DIR}>
   FILES ${CMAKE_CURRENT_BINARY_DIR}/helloworld_export.h)

However that still installs

1>-- Up-to-date: .../install/include/HelloWorld/HelloWorld.h
1>-- Up-to-date: .../install/include/helloworld_export.h

So it seems like the generator expressions aren’t working like the BUILD_LOCAL_INTERFACE still triggers in an install command.

Example.zip (1.8 KB)

1 Like

Try this:

install(
    TARGETS HelloWorld
    EXPORT HelloWorldExport
    FILE_SET HEADERS
    FILE_SET generated_export_headers
      DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}/HelloWorld"
)

It’s not clear to me why the default destination for HEADERS would be different than any other type-HEADERS file set though…

Ah, this is the problem. The path to HelloWorld.h from the basedir is HelloWorld/HelloWorld.h, so it is installed as such. Either use ${CMAKE_CURRENT_SOURCE_DIR} as a base directory or do:

Basically, the path from the containing BASE_DIR to the file is a “cononical name” for the header and should be included as such.

1 Like

Thanks for your reply. It did raise something I had not understood.

Why does

target_sources(HelloWorld PUBLIC 
   FILE_SET HEADERS 
   BASE_DIRS ${CMAKE_SOURCE_DIR}
   FILES HelloWorld.h)

work at all? I printed out the relevant CMake target properties after this call and get

get_target_property(HeaderDirs HelloWorld HEADER_DIRS)
get_target_property(HeaderSet HelloWorld HEADER_SET)

message(STATUS HeaderDirs=${HeaderDirs})
message(STATUS HeaderSets=${HeaderSets})
...
1>-- CMAKE_SOURCE_DIR=C:/.../src
1>-- HeaderDirs=C:/.../src 
1>-- HeaderSet=C:/../src/HelloWorld/HelloWorld.h

I sort of expected the following to work

target_sources(HelloWorld PUBLIC 
   FILE_SET HEADERS 
   BASE_DIRS ${CMAKE_SOURCE_DIR}
   FILES HelloWorld/HelloWorld.h)

However when I do this I get the following message, note the two levels of HelloWorld directories.

HeaderSet=C:/.../src/HelloWorld/HelloWorld/HelloWorld.h

The documentation states

FILES <files>...

If relative paths are specified, they are considered relative to CMAKE_CURRENT_SOURCE_DIR at the time target_sources() is called.

So if I have a file HelloWorld.h I assume that is regarded as a relative path.
If it is the implication seems to be that BASE_DIRS is totally ignored and CMAKE_CURRENT_SOURCE_DIR is always used.
If so I think the FILES <files>... should be amended to make extremely clear when it says

Each file must be in one of the base directories, or a subdirectory of one of the base directories.

that this only applies to absolute paths. IMHO ideally the FILES <files>... section would have two subsections one with a header files with relative or no path and a second with files with absolute path to make it clear that the behavior of the two wrt BASE_DIRS is totally different.

Another thing that the documentation is not very clear on is BASE_DIRS influence on the install locations of the header files. The documentation does mention that file sets can be installed with install(TARGETS) but doesn’t clarify how BASE_DIRS is involved.

If I am correct in my musings I am happy to have a submitting a PR with my stab at improving the documentation.

The paths you give to FILES are used to locate the files. You can think of the target_sources() command as using what you give there to work out the absolute path to each file. At generation time, CMake then looks at each file in turn and works out which of the BASE_DIRS that file sits under, and it removes the matching BASE_DIRS entry from the start of that file’s absolute path. What’s left is the path and file name that with be used when installing that file, treated as relative to the DESTINATION given in the install(TARGETS...) command.

Whether a file was originally specified as relative or absolute makes no difference to the above algorithm. Internally, it will effectively be converted to absolute anyway for the purposes of matching BASE_DIRS later.

In your examples, you use ${CMAKE_CURRENT_SOURCE_DIR} and ${CMAKE_SOURCE_DIR}. I’m not sure if you’re making errors or whether you did intend to use each of those. I would strongly discourage using ${CMAKE_SOURCE_DIR}. Imagine if some other project added yours as a vendored dependency. The ${CMAKE_SOURCE_DIR} is no longer going to be the same path relative to your project’s hierarchy as when the project is built standalone. I’m mentioning that because from what you posted, it was already not clear what path was intended by ${CMAKE_SOURCE_DIR}. It was only after downloading the zip file and looking at the project’s directory structure that I could work out what path was intended. For a command like target_sources(), try to avoid such “need to understand the whole project directory structure” variable usage. It will make the project easier to follow and less likely to have errors. I’ll also note that you have a similar problem with the usage of ${CMAKE_BINARY_DIR} instead of ${CMAKE_CURRENT_BINARY_DIR} in some places.

Focusing on some specific questions:

Yes, but perhaps $<BUILD_INTERFACE:...> and $<INSTALL_INTERFACE:...> might be special cases. The way BASE_DIRS is used, it doesn’t really make sense for its values to be different between build and install situations. The documentation could be updated to note that expressions like $<BUILD_INTERFACE:...> and $<INSTALL_INTERFACE:...> should not be used with BASE_DIRS.

I think this is more exposing that the way you’ve structured directories doesn’t quite match the expectations of file sets. Think of it as for each BASE_DIRS entry attached to a file set, it effectively adds a target_include_directories() call with that directory. That’s the first responsibility of BASE_DIRS. The second responsibility is constructing the relative path of files for installation, which I described in more detail above. Together, that implies that each header must be #include’d with a path under one of the BASE_DIRS.

For example, your original post specified the following:

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers 
   TYPE HEADERS
   BASE_DIRS 
        $<BUILD_INTERFACE:${CMAKE_BINARY_DIR}/HelloWorld> 
        $<INSTALL_INTERFACE:${CMAKE_BINARY_DIR}>
   FILES ${CMAKE_BINARY_DIR}/HelloWorld/helloworld_export.h)

That is capturing the confusion of what I think you’re falling victim to. The above is trying to say when building the project, there is no relative path to helloworld_export.h. Code would need to do a plain #include <helloworld_export.h> (angle brackets or quotes makes no difference here). However, once installed, the project is effectively trying to say (if $<INSTALL_INTERFACE:...> did actually work here) that code would need to use #include <HelloWorld/helloworld_export.h> instead. If that’s the intended path for the installed case, the project should use the exact same relative path when the project is being built too. It shouldn’t try to use different paths for the build versus install cases. I think that’s at the heart of your problems. If you address that, and avoid using ${CMAKE_SOURCE_DIR} or ${CMAKE_BINARY_DIR}, I think most of your issues will likely fall away and you’ll find no need to try to use generator expressions in BASE_DIRS.

You also might find it useful to be more explicit about your base directories by having a dedicated include directory in your directory structure. Part of your problem is that you’re trying to specify things that really belong in the parent directory of HelloWorld, but your project structure doesn’t really handle that well. Here’s what I recommend you try. I think you’ll then find things all fall out naturally and logically.

<topLevel>
   +-- HelloWorld
         +-- CMakeLists.txt
         +-- include
         |     +-- CMakeLists.txt
         |     +-- HelloWorld
         |           +-- CMakeLists.txt
         |           +-- HelloWorld.h
         +-- src
               +-- CMakeLists.txt
               +-- HelloWorld.cpp

HelloWorld/CMakeLists.txt:

add_executable(HelloWorld)

add_subdirectory(include)
add_subdirectory(src)

install(
    TARGETS HelloWorld
    EXPORT HelloWorldExport
    FILE_SET HEADERS
)

HelloWorld/include/CMakeLists.txt:

target_sources(HelloWorld
    PUBLIC
        FILE_SET HEADERS
        BASE_DIRS
            ${CMAKE_CURRENT_SOURCE_DIR}
            ${CMAKE_CURRENT_BINARY_DIR}
)
add_subdirectory(HelloWorld)

HelloWorld/include/HelloWorld/CMakeLists.txt:

generate_export_header(HelloWorld)

target_sources(HelloWorld
    PUBLIC
        FILE_SET HEADERS
        FILES
            HelloWorld.h
            ${CMAKE_CURRENT_BINARY_DIR}/helloworld_export.h
)

HelloWorld/src/CMakeLists.txt:

target_sources(HelloWorld
    PRIVATE
        HelloWorld.cpp
)

I haven’t tested any of the above, but hopefully it sketches out enough for you to follow the idea. I use this structure with my consulting clients on very large projects, and it handles the file sets nicely.

2 Likes

Thanks for the very detailed reply. To answer a couple of points

Generator expression in BASE_DIRS using things like $<BUILD_INTERFACE:...>

You write “Yes, but perhaps $<BUILD_INTERFACE:...> and $<INSTALL_INTERFACE:...> might be special cases”.
I very much agree with this in fact I noticed that if I use them in BASE_DIRS I get target properties like

1>   HelloWorld.INCLUDE_DIRECTORIES = "$<BUILD_INTERFACE:$<BUILD_INTERFACE:C:/.../build/HelloWorld>>;$<BUILD_INTERFACE:$<INSTALL_INTERFACE:C:/.../build>>"

I other words the wrapping of the BASE_DIRS with $<BUILD_INTERFACE> mentioned in the documentation happens even if the BASE_DIRS uses $<BUILD_INTERFACE:...> or $<INSTALL_INTERFACE:...>
I think either the wrapping should not be done in the case of BUILD_INTERFACE, INSTALL_INTERFACE and similar or there should be an error saying they are not allowed in BASE_DIRS along with a note to that effect in the documentation.

Using ${CMAKE_SOURCE_DIR}

Completely agree. The real world project does not do this. I used it in the example just to simplify things. Note in the real world project there are about 200 sibling project directories to HelloWorld

Using an include and src directory in the source tree.

I have worked on projects that do this and ones that put headers and implementation files in the same directory. I am ambivalent about this. In this case my original quest to use FILE_SET comes from a project that uses the headers and implementation files in the same directory approach. I can’t change overall structure of the source tree and there is no way the higher ups would accept such a massive restructure. I can only modify the existing CMake stuff.

" there is no relative path to helloworld_export.h"

What follows is the structure used by the real world project, I have to replicate the current behavior and file locations.
The current source/build tree does the following, the file HelloWorld.h does

#include "helloworld_export.h"
...

The helloworld_export.h is included in the build from ${CMAKE_CURRENT_BINARY_DIR} where it is generated so that directory needs to be in the build include path.
The install tree has to look like

<install_root>
    +- include
        +-- HelloWorld
            +-- HelloWorld.h
            +-- helloworld_export.h

Users of the install tree have the include directory in their include path, they do not have the HelloWorld directory in their include path. They do not #include the file helloworld_export.h directly they #include <HelloWorld/HelloWorld.h> which in turn does a #include "helloworld_export.h". This always works as the double quote style of include will always look for a file, in this case helloworld_export.h in the same directory as the file doing the #include. Doing a #include <helloworld_export.h> may do the same but it’s not a convention I’ve ever seen.

Anyway the following does replicate the existing tree and the CMakeLists.txt is definitely simpler

target_sources(HelloWorld PUBLIC 
   FILE_SET generated_export_headers  
   TYPE HEADERS
   BASE_DIRS ${CMAKE_BINARY_DIR}
   FILES ${CMAKE_CURRENT_BINARY_DIR}/helloworld_export.h)  

target_include_directories(HelloWorld PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>)

I was just trying to get rid of that target_include_directories by using the FILE_SET definition but it seem not to be possible.
I did just notice the documentation for generate_export_header does have an example very similar to what I am trying to do and it also has the target_include_directories probably for the same reason. It would be good if there were examples like this in the target_sources documentation.

Edited…
I just realized who you are. I have a copy of your book thanks so much again for replying to this. Maybe some of my struggles might help you understand what confuses us CMake mere mortals!
Note I had talked to a bunch of other CMake tinkerers I know and there was a general confusion about how BASE_DIRS works behind the scenes.

The reason this doesn’t work is because you’re trying to use #include "helloworld_export.h" instead of #include <HelloWorld/helloworld_export.h>. The former will only work when the header is in the same directory as the file that includes it, or if the directory where helloworld_export.h lives is on the header search path. You shouldn’t have that on the header search path, because that’s not what the installed case does. You worked around that by explicitly adding that directory to the header search path with target_include_directories(). You wouldn’t need to do that if you used #include <HelloWorld/helloworld_export.h>.

You’re welcome, and thanks for the feedback!

1 Like