How to install an external project?

Problem: Installing an external project

I am looking to have CMake clone and build a git repository as an ExternalProject, then package in the resulting binary files with the main project.

The documentation for ExternalProject mentions this:

Install Step Options:

The external project’s install rules are not part of the main project’s install rules, so if anything from the external project should be installed as part of the main build, these need to be specified in the main build as additional install() commands. The default install step builds the install target of the external project

In a sample project I have created an External Project that also uses CMake as its install system, it has an install command that looks like this:

# External project's install command
install (TARGETS extern_binary
         # Literally "SomeDestination", I don't know what directory to put here.
         RUNTIME DESTINATION "SomeDestination") 

The ExternalProject_Add command in my main project’s script looks like this:

# main project's ExternalProject_Add command
ExternalProject_Add(
	"extern" 
	GIT_REPOSITORY "[my repository on github here]"
	CMAKE_ARGS
		-DCMAKE_INSTALL_PREFIX:PATH=<INSTALL_DIR>
)

Unfortunately the executable file from the External Project is not getting packaged in by CPack. The executable file does not appear to be copied to anywhere the main project would know about, either.

Here’s my theories as to what’s happening:

  • ExternalProject_Add is not set up to cause the main project to run the External Project’s install step
  • The main project does run the external project’s install step, but it can’t find the directory containing the built binary file and I need to specify that somewhere
  • ExternalProject_Add alone can’t handle the install process automatically and I need to use something in addition to it

Here’s what I’ve found online:

  • (1) There is a Linux specific solution, using an INSTALL_COMMAND that involves autotools or make install may work but I need to support Windows which won’t have these tools in our case.
  • (2) This mailing list entry is similar but also seems to be Linux only
  • (3) This bug report looks promising but if I have my External Project print out the value of CMAKE_INSTALL_PREFIX it gives me build/extern-prefix. This is not what I expected, I suppose that I could do something like command cmake to go up a few directories and manually navigate to the right directory and copy the files there, but I’m hoping there’s a better way, because otherwise my External Project will have to be specially configured to work with my main build script.

Questions:

  • Is there a parameter I could provide to ExternalProject_Add that would cause my external project’s files to be automatically included in the install process?
  • If this isn’t possible, what is the least intrusive way that I can use install commands to accomplish this?

I am hoping there’s a builtin command I missed that would work seamlessly on all platforms (Windows, Mac, and Linux). Let me know if there’s something I missed here.

Thanks,

The <INSTALL_DIR> is just a placeholder to what you give in the INSTALL_DIR option in the ExternalProject_Add command.
Using a custom install dir should fix the problem. For example, here, I install the external project in the install subdirectory of the binary directory:

ExternalProject_Add(
	"extern" 
	GIT_REPOSITORY "[my repository on github here]"
	CMAKE_ARGS
		-DCMAKE_INSTALL_PREFIX:PATH=install
)

Usually, I want the dependencies to be installed alongside the main project, so I just forward the CMAKE_INSTALL_PREFIX:

-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}

Hope it helps,

Florian

1 Like

Hi Florian,

I have a couple more questions,

  • When you did -DCMAKE_INSTALL_PREFIX:PATH=install, can you give me an example of where this install directory is located?

I notice that CMAKE_INSTALL_PREFIX is “C:/Program Files (x86)/Tutorial” in the main project, which doesn’t make much sense to me. Just out of curiosity I tried forwarding this on to the external project thinking that maybe it would magically cause the external project’s install statements to be treated the same as install statements in the main project, but that didn’t seem to happen.

With a code block like this:

ExternalProject_Add(
	"extern" 
	GIT_REPOSITORY "[my repository on github here]"
	CMAKE_ARGS
		-DCMAKE_INSTALL_PREFIX:PATH=install
)

The binary from the external project ends up in the directory build\extern-prefix\src\extern-build\install and still does not get included in the NSIS installer or the CPack directory in the main project.

  • Do I need to add additional install commands in the main project on each target of the external project?

I think this would probably work but it seems a bit hokey. I’m not sure why the install commands from the external project can’t “transfer” over to the main project.

Thanks again,

The CMAKE_INSTALL_PREFIX cache variable tells CMake where to install your project. By default it points to C:/Program Files (x86)/${PROJECT_NAME}.
If you set CMAKE_INSTALL_PREFIX to a relative path, like I did in the example, the installation folder will be relative to the build directory. So in the case of -DCMAKE_INSTALL_PREFIX:PATH=install, the installation directory will be relative to the build directory of the external project, which is build\extern-prefix\src\extern-build\.

Do I need to add additional `install` commands in the main project on each target of the external project?

No. Each project should be self managed, installing its own targets.

Let’s say you have your main project called Main that have a dependency on a library called Dep.

  • The Dep library should install everything needed by users of the library using the install command.
  • The Main project should find the Dep library using a call to find_package.

If you want to automate the building of Dep for the developers of Main, you should use the Superbuild pattern. The idea behind the superbuild is to create a specific CMake project that will

  • build the project dependencies using ExternalProject commands
  • build the main project itself using ExternalProject
    • forwarding the dependencies build locations using the ${dependency_name}_DIR variables
    • adding an explicit dependency to the corresponding external projects using the DEPENDS keyword.

In your case, it might look like this:

ExternalProject_Add(
	Dep
	GIT_REPOSITORY "[Dep github here]"
	CMAKE_ARGS
		-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Dep-install
)

ExternalProject_Add(
	Main
	GIT_REPOSITORY "[Main github here]"
	CMAKE_ARGS
		-DDep_DIR:PATH=${CMAKE_CURRENT_BINARY_DIR}/Dep-install
		-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Main-install
 	DEPENDS Dep
)

Regards,

Florian

2 Likes

Hi Florian,

I have went ahead and converted my example to use the SuperBuild format. I have two ExternalProject_Add statements similar to yours now, the superbuild concept makes a lot more sense now, thanks.

I still have a couple more questions though,

  • At first I tried just include(CPack) in my “root” build script that contains both of these ExternalProject_Add statements, it seemed promising at first, but the NSIS installer and the CPack directory contained nothing at all after I ran package from the root visual studio solution

  • The next thing I tried was having the Main external project be responsible for packaging, I included CPack in that project’s CMakeLists.txt instead

This seems like the right way to do it and I think I’ve seen other superbuilds do this.

Though, I don’t quite understand how the Main project is made aware of Dep's targets.

  • Just passing in -DDep_DIR:PATH=${CMAKE_CURRENT_BINARY_DIR}/Dep-install by itself had no effect, though I didn’t think it was going to be that easy. I was also hoping that DEPENDS would help with finding targets in the Dep, but this doesn’t seem to be the case either.
  • Attempting to do find_package(Dep) did not work either, but I figured it wouldn’t be that easy either, because find_package isn’t looking for binary files, it’s looking for findDep.cmake in CMAKE_MODULE_PATH

So… based on that, I concluded that the next step was to create findDep.cmake in Main.

  • I am not sure if this was the right thing to do, because intuitively to me it seems like Dep should provide this find script since only it should know its targets, right?
  • Or is the idea that Main only takes the targets it needs for its own uses from Dep? (whereas a Dep find script would provide everything including libraries that Main might not need, like if Dep was a giant library like OpenCV and you only needed part of it)

My find script looks like this, to be honest I’m not 100% sure what I’m supposed to do here, it kind of seems like I’m redefining the targets of Dep (which doesn’t seem like a very clean way to do it).

#findSUPERBUILD_EXTERN.cmake (for me, the Dep project is called SUPERBUILD_EXTERN)
message("Start of find script for the extern project.")

# note that SUPERBUILD_EXTERN_DIR is a cache variable for this project
find_program(
            EXTERN_BINARY 
            "extern_binary"
            ${SUPERBUILD_EXTERN_DIR}]
        )

message("EXTERN_BINARY IS " ${EXTERN_BINARY})

set(SUPERBUILD_EXTERN_EXECUTABLE ${EXTERN_BINARY})

With this find script, it locates extern_binary, but this binary is still not packaged when I build the PACKAGE project in Main.

Thanks again, I appreciate all your help! Let me know if you have any more ideas,

Before going into the Packaging step, you need to have a fully working superbuild, so let’s focus on that first.

The find_package command

The find_package command have two different modes: Module and Config.

  • Module mode will search for a FindDep.cmake file in ${CMAKE_MODULE_PATH},
  • Config mode with search for a DepConfig.cmake file in the ${Dep_DIR} directory.

Usually, Module mode is used when you try to search for a system library or utility, and Config mode when you try to search a library that you built. In your case, you want to find a lib that you compiled yourself, so you want to use the Config mode.

Writing the DepConfig.cmake file

Every CMake project that is meant to be used by other CMake projects should “export” its targets. This is done like this:

install(TARGETS Dep
        EXPORT DepTargets
        LIBRARY DESTINATION lib
        ARCHIVE DESTINATION lib
        RUNTIME DESTINATION bin
        PUBLIC_HEADER DESTINATION include
        INCLUDES DESTINATION include
)

install(EXPORT DepTargets
        FILE DepTargets.cmake
        NAMESPACE Dep::
        DESTINATION lib/cmake/Dep
)

These install calls will make sure the Dep target is installed (its binaries, public headers, etc.). It will also generate a “DepTargets.cmake” file that will contain all the targets that have been installed. These targets won’t be the original targets, but what we call IMPORTED targets. They will not refer to the sources, but directly to the binaries that have been installed.

In order to be used by the find_package command, the DepTargets.cmake file should be included in the DepConfig.cmake file used by the find_package. We will use the following commands to generate this file:

configure_file(${CMAKE_SOURCE_DIR}/cmake/DepConfig.cmake.in DepConfig.cmake @ONLY)
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/DepConfig.cmake"
        DESTINATION lib/cmake/Dep
)

where the DepConfig.cmake.in file content is as follows:

include("${CMAKE_CURRENT_LIST_DIR}/DepTargets.cmake")

Making the superbuild work

Now that we have written/generated all the necessary files, the superbuild needs to be adjusted in order to allow the Main project to find its dependency. The Dep_DIR variable should point to the folder containing the DepConfig.cmake file, which gives:

ExternalProject_Add(
	Dep
	GIT_REPOSITORY "[Dep github here]"
	CMAKE_ARGS
		-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Dep-install
)

ExternalProject_Add(
	Main
	GIT_REPOSITORY "[Main github here]"
	CMAKE_ARGS
		-DDep_DIR:PATH=${CMAKE_CURRENT_BINARY_DIR}/Dep-install/lib/cmake/Dep
		-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Main-install
 	DEPENDS Dep
)

With all of the above, your project should install correctly, and build using the superbuild pattern.

Packaging

Now that your project builds and installs correctly, you should be able to package it by doing include(CPack) in the Main project.

Best regards,

Florian

2 Likes

Hi Florian,

The flow of control of this operation is… surprising to say the least, I don’t think I would have figured this out without your help!

I have went ahead and made the changes you suggested, but unfortunately it still does not work, here are my questions:

I’m not 100% sure where this code should go:

install(TARGETS Dep
        EXPORT DepTargets
        LIBRARY DESTINATION lib
        ARCHIVE DESTINATION lib
        RUNTIME DESTINATION bin
        PUBLIC_HEADER DESTINATION include
        INCLUDES DESTINATION include
)

install(EXPORT DepTargets
        FILE DepTargets.cmake
        NAMESPACE Dep::
        DESTINATION lib/cmake/Dep
)

I assumed that both of these should go in Dep’s CMakeLists.txt, I put them there, more or less as you had them, but since the external project only has a single executable, the first command ended up looking more like this:

install(TARGETS Dep
        EXPORT DepTargets
        RUNTIME DESTINATION bin
)

I’m not sure why DepConfig.cmake.in needs to be configured

It seems like the output after configuration matches the contents of DepConfig.cmake.in exactly, there’s no @ variables in DepConfig.cmake.in, and it’s using the @ONLY parameter.

I’m assuming that you just included the configure_file step because this is a standard step in the “Superbuild template” and to show me somewhere that I can expand on it later, is that correct?

How do I install the imported target in Main?

Unfortunately there still seems to be something missing here.

  • At first I assumed that just by using find_package, CPack would be smart enough to recognize that it needs to pull in Dep’s targets and package them as well.
  • I did not get any errors from CMake from the find_package command and I used print statements to verify that the config file was executing.
  • Unfortunately it seems like find_package does not automatically install Dep’s targets, when I tried to package, the package contained only Main’s binary and not the external project’s binary.
  • I was able to verify that the target was getting imported into the Main project using if(TARGET extern_binary); it appears that the target import was successful, I printed out some property values of the target and it seems to be a healthy imported target, with valid paths to executables that exist (see source code below).

After that I tried adding an install command to Main’s CMakeLists.txt, to install the imported target. Unfortunately, when I try that, it fails with the error message:

install TARGETS given target “extern_binary” which does not exist.

This error message is very confusing because if(TARGET extern_binary) is true, and the target clearly has properties. I have no idea what this error message is really trying to say.

Wild guess: it’s not a TARGET to Main?

I took a wild guess and assumed that maybe from the perspective of the Main project, extern_binary is no longer a TARGET and is actually a PROGRAM.

  • I didn’t really expect this to work because the generated SUPERBUILD_EXTERNTargets.cmake file is clearly using add_executable(extern_binary IMPORTED), which makes a TARGET.
  • I didn’t have much success with this route either; I tried using both extern_binary and the path extracted from extern_binary's IMPORTED_LOCATION_DEBUG
  • The documentation also says that it’s rarely needed to extract the location out of an imported target, but I don’t see a good example of how to have Main use this imported target…

Here’s an excerpt from Main’s CMakeLists.txt that handles the external project
(this does not include my experimentation with attempting to have Main add it as a PROGRAM)

# Note: I removed the Namespace from install(EXPORT in the external dependency,
# on the off chance that I was running into some obscure bug in CMake 3.13
# because all of the examples online didn't use namespaces.
#
# Removing the namespace had no effect on the outcome.
message("SUPERBUILD_EXTERN_DIR is " ${SUPERBUILD_EXTERN_DIR})

find_package(SUPERBUILD_EXTERN)

if(TARGET extern_binary)
    message("extern_binary exists as a target!")
else()
    message("target extern_binary does not exist")
endif()

# I tried making the name of the export different from the exported
# target itself, but it didn't seem to have any effect, I think you can
# disregard this.
if(TARGET extern_export_binary)
    message("extern_export_binary exists as a target!")
else()
    message("target extern_export_binary does not exist")
endif()

# just blank
#message("extern_binary is " ${extern_binary})

get_property(extern_location
            TARGET extern_binary
            PROPERTY IMPORTED_LOCATION)

get_property(isImported
            TARGET extern_binary
            PROPERTY IMPORTED)

get_property(configs
            TARGET extern_binary
            PROPERTY IMPORTED_CONFIGURATIONS)

get_property(extern_location_debug
            TARGET extern_binary
            PROPERTY IMPORTED_LOCATION_DEBUG)

message("extern_location is " ${extern_location})
message("isImported is " ${isImported})
message("configs is " ${configs})
message("extern_location_debug is " ${extern_location_debug})

install (TARGETS main_binary 
         RUNTIME DESTINATION "DirectoryForCmakeSuperbuild_Main")

install(TARGETS extern_binary
        RUNTIME DESTINATION "DirectoryForCmakeSuperbuild_MainExtern")

Here’s what it outputs:

3>  SUPERBUILD_EXTERN_DIR is [my build directory]/extern-install/lib/cmake/Extern
3>  extern_binary exists as a target!
3>  target extern_export_binary does not exist
3>  extern_location is 
3>  isImported is TRUE
3>  configs is DEBUG
3>  extern_location_debug is [my build directory]/extern-install/DirectoryForCmakeSuperbuild_Extern/extern_binary.exe

Thanks again,

Hi,

I’m not 100% sure where this code should go

You are right, this code should be part of Dep main CMakeLists.txt.

I’m not sure why DepConfig.cmake.in needs to be configured

In you case, this is not really needed. But usually, projects want to expose some configuration variables to the installation, so a configure step is needed.

How do I install the imported target in Main?

You should not, I guess. I think CMake forbid the installation of IMPORTED targets, that is why you have this error.

I think that maybe the packaging step should go to the superbuild script. What happen if you include CPack there ?

Florian

1 Like

Hi Florian,

I think that maybe the packaging step should go to the superbuild script. What happen if you include CPack there ?

I think this makes a lot of sense, but I actually attempted this before putting CPack in Main.

  • If I try this I end up with nothing in the package and the cpack folder is completely empty. I tried it again just now and I got the same result.

  • I think that’s because the SuperBuild CMakeLists.txt doesn’t have anything except ExternalProject_Add commands:

# The Superbuild project's CMakeLists.txt
cmake_minimum_required (VERSION 2.6)

project(SUPERBUILD_ROOT)

include(ExternalProject)

include(CPack)
set(CPACK_GENERATOR NSIS)


ExternalProject_Add(extern
                    GIT_REPOSITORY "[extern's git repository]"
                    CMAKE_ARGS
                        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/extern-install
)

ExternalProject_Add(Main
                    GIT_REPOSITORY "[main's git repository]"
                    CMAKE_ARGS
                        -DSUPERBUILD_EXTERN_DIR:PATH=${CMAKE_CURRENT_BINARY_DIR}/extern-install/lib/cmake/Extern
                        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Main-install
                    DEPENDS 
                        extern
)

If it is necessary to put install commands in this, then, well, it seems to me that we’d end up in the exact same situation that Main is in, where we can’t install imported targets.

Let me know if you have any other suggestions, thanks,

EDIT:

Yep, looking at the source, the install of imported targets is actively forbidden in CMake 3.17, (1), CMake 3.13.5 (which is what I’m using) does something… else (2)?

Hi,

It seems to be more complicated than I thought. I found this: https://cmake.org/pipermail/cmake/2011-May/044344.html

To resume, the CPACK_INSTALL_CMAKE_PROJECTS variable is a list containing, for each project you want to install, its

  • Build directory,
  • Project Name,
  • Project Component,
  • Install Directory

So, in your case, you should do something like this:

set(CPACK_INSTALL_CMAKE_PROJECTS
"${CPACK_INSTALL_CMAKE_PROJECTS};${Dep_DIR};Dep;ALL;/")

set(CPACK_INSTALL_CMAKE_PROJECTS
"${CPACK_INSTALL_CMAKE_PROJECTS};${Main_DIR};Main;ALL;/")

Best,

Florian

1 Like

Hi Florian,

Thanks! The CPACK_INSTALL_CMAKE_PROJECTS was the thing that I was missing, this was quite a journey!

I did some experimentation and it seems like that’s about all I needed actually; I was able to remove all of the target imports and exports and the packaging still worked.

Here’s what my files looked like:

Root / Superbuild CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project(SUPERBUILD_ROOT)

include(ExternalProject)

# ----------------------------------------------------------------------------------------------------------------------
# Set up our external projects
# ----------------------------------------------------------------------------------------------------------------------

ExternalProject_Add(extern
                    GIT_REPOSITORY "[extern's git repository]"
                    CMAKE_ARGS
                        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/extern-install
)

ExternalProject_Add(Main
                    GIT_REPOSITORY "[main's git repository]"
                    CMAKE_ARGS
                        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/Main-install
                    DEPENDS 
                        extern
)

# ----------------------------------------------------------------------------------------------------------------------
# Read the "binary directories" of both external projects
#
# CMake calls the build\extern-prefix\src\extern-build a "Binary directory", but it's really the directory where the VS Solution is, etc.
# I'd call these "build directories", to be honest...
# ----------------------------------------------------------------------------------------------------------------------

unset(BINARY_DIR)

#This function writes to a variable named BUILD_DIR... okay sure?
#It'll get clobbered twice so we have to save the result somewhere else.....
ExternalProject_Get_property(extern BINARY_DIR) 
set(externBuildDirectory ${BINARY_DIR})
message("extern's build directory is " ${externBuildDirectory})

unset(BINARY_DIR)

ExternalProject_Get_property(Main BINARY_DIR)
set(mainBuildDirectory ${BINARY_DIR})
message("Main's build directory is " ${mainBuildDirectory})

unset(BINARY_DIR)

# ----------------------------------------------------------------------------------------------------------------------
# Write these paths to CPACK_INSTALL_CMAKE_PROJECTS so that CPack will know to visit these projects for packaging
# ----------------------------------------------------------------------------------------------------------------------

# https://cmake.org/cmake/help/latest/module/CPack.html?highlight=cpack_install_cmake_projects#variable:CPACK_INSTALL_CMAKE_PROJECTS
#                       project build dir       proj name   build all components    "Directory" (??)
set(externProjectInfo   ${externBuildDirectory} "extern"    "ALL"                   "/")
set(mainProjectInfo     ${mainBuildDirectory}   "Main"      "ALL"                   "/")

# both of these go in one list
set(allProjectInfo ${mainProjectInfo} ${externProjectInfo})

message("CPACK_INSTALL_CMAKE_PROJECTS before append is " ${CPACK_INSTALL_CMAKE_PROJECTS})

# I think this variable has to be set before CPack is included.
set(CPACK_INSTALL_CMAKE_PROJECTS ${allProjectInfo})

message("CPACK_INSTALL_CMAKE_PROJECTS after append is " ${CPACK_INSTALL_CMAKE_PROJECTS})

# ----------------------------------------------------------------------------------------------------------------------
# Now include CPack (must be last)
# ----------------------------------------------------------------------------------------------------------------------

include(CPack)
set(CPACK_GENERATOR NSIS)

Extern (AKA Dep)'s CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project(SUPERBUILD_EXTERN)

message("Hello from the external repository's CMakeLists.txt!")

# Creates the executable with the listed sources and adds sources to the Solution Explorer
# (but you won't see extern.c in the root (superbuild) solution)
add_executable (extern_binary extern.c)

install (TARGETS extern_binary 
        RUNTIME DESTINATION "DirectoryForCmakeSuperbuild_Extern"
        )

Main’s CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project(SUPERBUILD_MAIN)

message("Hello from the main repository's CMakeLists.txt!")

# Creates the executable with the listed sources and adds sources to the Solution Explorer
# (but you won't see main.c in the root (superbuild) solution)
add_executable (main_binary main.c)

install (TARGETS main_binary 
         RUNTIME DESTINATION "DirectoryForCmakeSuperbuild_Main")

I suppose I’ll also put my github repositories here too, in case it helps anyone else:

One thing I’m wondering about though,

If it’s not possible to install using imported projects, what is the purpose of importing targets? Is it just for programs that don’t need to be packaged?

1 Like

Great !

I am not really sure why we can’t install imported targets. My guess is that it uses the philosophy of *nix systems: If the dependence is a static lib, you don’t need to package it, and if it is a shared lib, you need to package it on its own…

Best regards,

Florian