Integrate large repositories into your project.

Code
,
1

Hi there,

In my project, I need to work with wxWidgets and OpenCV to implement GUI functionalities. Right now, I am using the FetchContent module to integrate the libraries mentioned, and, as you may know, they are quite big libraries so the configuration and building time is increasing for the project.

I was wondering if there is any pattern in CMake to work with big repos.

Is it better to install the libraries or just continue with the FetchContent module to integrate them? I just live with the added time for compilation.

2

For such large and widely available libraries, I think find_package and expecting the developer environment provide them is a far better solution.

Cc: @craig.scott

1 Like
3

I believe the same. The issue is that my manager does not want to install any library on the computer. Don’t ask me why.

Now because I am fetching these two libraries, the building time for our small device is huge.

4

All I can suggest is to present the time factor to your manager(s) and see if a process for blessing such important libraries is worth the time savings.

5

Apart from installing them locally, you could store the libraries in some artifact management system (or a simple web server, or your version control system).

6

I always understood that having the libs and stuff in a repository was not so good. That would make the repository so big, and the cloning time longer.

Do you recommend having the lib or DLL files and the header files in the repository? If there is not an option of installing the library locally.

7

You can store them on some static location and have your CMake configure step download and extract them for you automatically.

8

Thanks for the tip, Ben.
The thing I see as an issue to this approach is on the CI/CD part when you use a cloud service, or at least I am knowledgeable in that kind of configuration.

Do you know any example that I can take a look at?
I think you can do this and store the libs in another repository as mentioned before, but in the end, you have to clone the repository in your project so the cloning time will be still long, but your project repository is not polluted with compiled files.

9

Do you have a static web host you can toss the builds up onto? That’s all you really need to store a tarball that you download (if needed).

1 Like
10

The VCS solution depends on which VCS you use. In case of git you can add those files preferably in a separate repository which you can clone with --depth 1 to get only the version you need. Or just download if directly by http (depending on the git repo server you use).
You could also use LFS.

In case of a CI system you can simply split the build into mulitple pipelines and access the artifacts built by your wxWidgets and OpenCV pipeline.
You can even use those artifacts for your local build if your able to download them.

11

Hi Josef,

Thank you for your comment, just checking the git lfs seems a good option, also to work with data. In that aspect, that solution may work for decreasing the compilation time.

For CI, I am using the one provided in GitLab, so I will deal with the default windows image given by them, and the pipeline is already stuff that I put in place but using the windows image requires to do the creation and set up in each job, I am passing the build folder between the jobs. As this topic is not really CMake-related, I will stop it here.

Thanks for all the help!

12

I use add subdirectory opencv which contains a git submodule of opencv. This work quite well, especially with ccache eliminating build times for the parts that do not change.

Another option is to append the cmake modules path and place a findopencv.cmake or opencvmodule in it which points to home/user/local/opencv/inlcude,lib etc. Then set opencv install prefix dir, build it and install.

13

Hey, I have a similar issue, but I am still going with fetch content, just additionally using ccache. If you use ccache you can get it working even in your CI system (provided it supports caches in general).

14

Thanks for the info @eri0 I will have a look into this, seems like a good solution for the CI.

Edit: It seems that the Microsoft Visual C++ compiler, the one I am using because here at the company we work with windows (Yes, I know) is not compatible with CCACHE

15

MSVC does work with sccache.

2 Likes