FetchContent a directory but add a subdirectory

I have some code using git submodules that is:

  1. Working on a full checkout of the highs library, let’s say in deps/highs.
  2. Calling add_subdirectory(deps/highs/src).

The deps/highs/src/CMakeLists.txt file contains references to ${HIGHS_SOURCE_DIR}, which is deps/highs, i.e. the parent directory.

Is there a way to accomplish the same functionality with FetchContent? I.e. download everything but add only src subdirectory. Because using SOURCE_SUBDIR src in FetchContent_Declare will change ${HIGHS_SOURCE_DIR} to build/_deps/highs/src, won’t it? I have tried that and it doesn’t seem to do the work.

Thanks!

Can you just set it via -D? Skipping the top-level logic doesn’t seem like it’s something the project really supports.

1 Like

Hi @ben.boeckel,

I’m sorry but I don’t understand your answer. Could you please elaborate a bit more on it?

Thanks!

With FetchContent, add -DHIGHS_SOURCE_DIR=<SOURCE_DIR> to the CMAKE_ARGS.

1 Like

OK, thanks!

I could hardcode that, yes, but I would also need to hardcode HIGHS_BINARY_DIR, and I wouldn’t be sure yet I would be missing something else.

Yeah, that’s a likely one. I’d take a look at the CMakeLists.txt to see what logic you might be skipping over at least.

1 Like

Sorry again, @ben.boeckel , but I don’t see how setting -DHIGHS_SOURCE_DIR can help here.

  • FetchContent will set HIGHS_SOURCE_DIR to the root folder.
  • If I tell CMake that HIGHS_SOURCE_DIR is the src folder within root, then, when it configures Highs, and it gets to src/CMakeLists.txt, the code in there will expect HIGHS_SOURCE_DIR to point to the root folder.

Couldn’t I use just FetchContent_Declare (or ExteranalProject_Add) plus add_subdirectory("${HIGHS_SOURCE_DIR}/src")?

Why would it do that? These variables usually come from a project(HIGHS) call.

1 Like

Oh, sure, I wasn’t seeing that. FetchContent will set highs_SOURCE_DIR but not HIGHS_SOURCE_DIR. You’re right, thanks!

So is your suggestion then:

  • FetchContent with SOURCE_SUBDIR pointing to src, plus
  • Setting HIGHS_SOURCE_DIR to `${highs_SOURCE_DIR}/…" and “HIGHS_BINARY_DIR” to “${highs_BINARY_DIR}/…”?

It seems the code below is working.

It uses your suggestion of setting HIGHS_SOURCE_DIR (BTW, the project I am modifying, which was managing dependencies via git submodules, was already doing that).

But it also uses:

  • FetchContent_Declare, to declare dependencies,
  • FetchContent_Populate, to download dependencies and set highs_SOURCE_DIR and higs_BINARY_DIR,
  • set(HIGHS_SOURCE_DIR) and set(HIGHS_BINARY_DIR), the two variables that are accessed by src/CMakeLists.txt and that have to point to the parent dir of src, and
  • add_subdirectory(src), to process only src subdir.
FetchContent_Declare(highs
    GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git
    GIT_TAG "45a127b78060942721f75f46a04b274c2bb963d8"
)
FetchContent_Populate(highs)
set(HIGHS_SOURCE_DIR "${highs_SOURCE_DIR}")
set(HIGHS_BINARY_DIR "${highs_BINARY_DIR}")
add_subdirectory("${highs_SOURCE_DIR}/src" "${highs_BINARY_DIR}/src")

I might try doing the same with:

FetchContent_Declare(highs
    GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git
    GIT_TAG "45a127b78060942721f75f46a04b274c2bb963d8"
    SOURCE_SUBDIR src
)
set(HIGHS_SOURCE_DIR "${highs_SOURCE_DIR}")
set(HIGHS_BINARY_DIR "${highs_BINARY_DIR}")
FetchContent_MakeAvailable(highs)

The first of the solutions from my last post, the one using add_subdirectory, has the problem that the subfolder is treated like a project’s folder. So, for example, if I build my project with warnings as errors, and the subproject raises some warnings, those will be treated as errors of my project.

Fortunately, it can be avoided using CMake 3.25’s add_subdirectory(... SYSTEM).

Avoid calling FetchContent_Populate() directly. I plan to eventually deprecate that. Call FetchContent_MakeAvailable() instead.

Note also that you can add the SYSTEM keyword to FetchContent_Declare() starting with CMake 3.25 as well.

1 Like

That’s the wrong order. The highs_SOURCE_DIR and highs_BINARY_DIR variables won’t be populated until after the call to FetchContent_MakeAvailable().

EDIT: Oh, I see that reordering would be a problem in this case. You need HIGHS_SOURCE_DIR and HIGHS_BINARY_DIR to be populated before the downloaded source has been loaded. You could let FetchContent handle the downloading only, but take control of the add_subdirectory() part by setting SOURCE_SUBDIR to a non-existent subdirectory. That will prevent FetchContent_MakeAvailable() from calling add_subdirectory(). The main advantage to using FetchContent_MakeAvailable() is for the dependency provider and find_package() integrations, which might not be important to you right now, but may be in the future or for other consumers of your project.

1 Like

Many thanks for your comments @craig.scott.

Are you then proposing something like this?

FetchContent_Declare(highs
    GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git
    GIT_TAG "45a127b78060942721f75f46a04b274c2bb963d8"
    SOURCE_SUBDIR this-directory-does-not-exist
)

# FechContentMakeAvailable will set
# highs_SOURCE_DIR to _deps/highs-src/this-directory-does-not-exist, and
# highs_BINARY_DIR to _deps/highs-build/this-directory-does-not-exist
FetchContent_MakeAvailable(highs)

# Properly setting
# highs_SOURCE_DIR to _deps/highs-src/src, and
# higs_BINARY_DIR to _deps/highs-build/src
set(highs_SOURCE_DIR "${highs_SOURCE_DIR}/../src")
set(highs_BINARY_DIR "${highs_BINARY_DIR}/../src")

# Now carry on doing what we were doing with FetchContent_Populate
set(HIGHS_SOURCE_DIR "${highs_SOURCE_DIR}/..")
set(HIGHS_BINARY_DIR "${highs_BINARY_DIR}/..")
add_subdirectory("${highs_SOURCE_DIR}/src" "${highs_BINARY_DIR}/src" SYSTEM)

OK, I had it working with the following code:

FetchContent_Declare(highs
    GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git
    GIT_TAG "45a127b78060942721f75f46a04b274c2bb963d8"
    SOURCE_SUBDIR this-directory-does-not-exist
)
FetchContent_MakeAvailable(highs)
set(HIGHS_SOURCE_DIR "${highs_SOURCE_DIR}")
set(HIGHS_BINARY_DIR "${highs_BINARY_DIR}")
add_subdirectory("${highs_SOURCE_DIR}/src" "${highs_BINARY_DIR}/src" SYSTEM)

@craig.scott Hello sir! Is there a workaround for the case where we want to specify SOURCE_DIR? Specifically, all I want FetchContent to do is download my desired repository to a desired location. I want to then handle the add_subdirectory call myself from there.

I’m using SOURCE_DIR to designate the desired download location of my packages, because DOWNLOAD_DIR is (apparently) only used for URL-based FetchContent calls.

If I can’t specify SOURCE_DIR, then I can’t actually control where my code gets downloaded, but I also don’t want FetchContent_MakeAvailable invoking the dependency’s CMakeLists.txt.

I can try to get fancy, and move the content to my desired location, but CMake still populates the target.

add_library cannot create target "my_fetched_target" because another target
  with the same name already exists.  The existing target is a static library
  created in source directory
  "/home/path/to/build/preset/_deps/my_package-src".  See
  documentation for policy CMP0002 for more details.

Plus it’s ugly:

# this is a FetchContent-based dependency
include(FetchContent)

# configure find_package calls
set(MY_CONTENT_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/package")

set(FETCHCONTENT_QUIET FALSE)

# To preserve install behavior transitively, we add the subdirectory instead of actually finding the package
# declare dependency source
FetchContent_Declare(
	my_package
	GIT_REPOSITORY "${FETCHCONTENT_GITLAB_ORIGIN}/my_repo.git"
	GIT_TAG "main"
	GIT_SHALLOW    TRUE
	GIT_PROGRESS   TRUE
#        OVERRIDE_FIND_PACKAGE
	EXCLUDE_FROM_ALL
)

# Download the content without making it available to the build.
# We will use add_subdirectory for that
set(my_package_POPULATED TRUE)
FetchContent_MakeAvailable(my_package)

if(IS_DIRECTORY "${my_package_SOURCE_DIR}")
	# Only try to move source directory if it exists
	file(REMOVE_RECURSE "${MY_CONTENT_SOURCE_DIR}")
	file(RENAME
		"${my_package_SOURCE_DIR}"
		"${MY_CONTENT_SOURCE_DIR}"
		RESULT rename_result
	)

	message(STATUS "Moved source directory: ${rename_result}")
endif()

add_subdirectory(package)

Edit: Seems like using FetchContent_Populate with the supported arguments from FetchContent_Declare gets me what I want. Huzzah!

Lots to unpack there. First up, FetchContent_Populate() shouldn’t be used directly from projects. I won’t go into the various reasons why (its history isn’t short, and it will distract this topic from its central focus). FetchContent_Populate() is not the path to happiness. You should be aiming to use FetchContent_MakeAvailable() instead.

What you’re after is setting SOURCE_DIR to where you want the source to be located, then set SOURCE_SUBDIR to a non-existent directory (these are in your call to FetchContent_Declare()). That will achieve your stated goal of downloading the content without FetchContent_MakeAvailable() calling add_subdirectory() on it.

Don’t try to move things after FetchContent has downloaded them. FetchContent expects things to stay where it put them, and you’re going to run into a variety of problems if you try to move things around. FetchContent uses time stamps to work out what steps it has done, and if you move things, you take things out from under its feet. If you really must specify where the source should be located, SOURCE_DIR is the right way to do that.

Normally, you should just let FetchContent put the source wherever it wants. You can always obtain the location of where it put it using the <lowercaseDepName>_SOURCE_DIR variable. This is usually preferable to trying to manually manage where to put things.

I really wish I could read.. I’d conflated SOURCE_DIR with SOURCE_SUBDIR. What you’ve described works perfectly. Thank you, sir!