cmake, ccache, CCACHE_BASEDIR: wrong paths in compiler error messages

Hi,

Please let me summarize what we have here. We have projects with some hunderds CMakeLists.txt. The “main” one typically use several add_subdirectory directives for the use libraries in form of:

  add_subdirectory(${path_lib_a} liba)

For each, cmake creates a subdirectory (such as here liba) in the build directory. Normally cmake uses absolute paths, so compiler error messages contain absolute paths as well.

Now we are also using ccache. Since we use the same libs in many places, of course we want to share the ccache results. For multiple builddirs I was not able to get it working, but at least for the case of multiple sourcedirs that have there builddirs below. This is useful if several people work on same code (shared ccache) or if you have several checkouts, for example for similar variants or source code branches. In most of these cases, many libraries are the same, so caching in theory should be very effective.

Since ccache also compares compiler arguments of course and it’s include arguments include absolute paths, ccaching across source trees does not work, so we have to set CCACHE_BASEDIR.

Technically, our CMakeLists.txt create wrapper files used as CMAKE_CXX_COMPILER_LAUNCHER in BUILDDIR with "export CCACHE_BASEDIR=$TOPSRCDIR" and then calling essentially "ccache $CXX "$@"". Now ccache works fine across source trees. At the end of the mail some cmake excerpt in case it is useful.

But unfortunately now IDEs (like Vim, QTCreator…) are unable to jump to compiler errors, because now the paths are relative to the directory where the compiler (wrapper) is called. Normally the IDEs know this from make output:

  make[2]: Entering directory '/home/sdettmer/work/tisc-src/build/cmake-build-debug-1810'

but unfortunately for “add_subdirectory”, cmake uses some own logic
visible only when running make VERBOSE=1 (or I think make -w), which looks like:

  make[2]: Entering directory '/home/sdettmer/work/tisc-src/build/cmake-build-debug-1810'
  [ 14%] Building CXX object....
  cd /home/sdettmer/work/tisc-src/build/cmake-build-debug-1810/libA
    && /home/sdettmer/work/tisc-src/build/cmake-build-debug-1810/ccache-CXX
    /opt/bt/cross/i586-pc-linux-gnu/gcc-4.4.3/usr/bin/i586-pc-linux-gnu-g++
    ...
    -o CMakeFiles/libA.dir/home/sdettmer/work/tisc-src/.../libA/src/File1.cpp.o
     -c /home/sdettmer/work/tisc-src/.../libA /src/File1.cpp

Compiler errrors now are not relative to the path told by make (here cmake-build-debug-1810), but relative to the "cd .../libA" – which is normally not even visible:

  In file included from ../../..[...]/libA/File1.cpp:10: ....

Now the IDE looks for path relative to cmake-build-debug-1810, because it does not know about the "cd [...]cmake-build-debug-1810/libA" and thus does not find the file!

We discussed this in the team, I searched the web for several hours now, but I don’t find any solution. Due to the absolute paths requirement of cmake I think I must use CCACHE_BASEDIR, but when I do, IDEs don’t find the files. I didn’t find a way to make cmake output Entering directory lines. Since we are probably not the only ones who use cmake and ccache, we probably doing it wrong and miss an important point.

How is ccache supposed to be used correctly? I saw many examples in the internet, but none seem to correctly work with CCACHE_BASEDIR, and without CCACHE_BASEDIR cache is not efficient, almost useless and shared cache even completely useless (never hits).

Any help appreciated!

As only workaround we could find, I now manually add:

  echo "make[77]: Entering directory '$(pwd)'"

in the ccache-wrapper. This helps a bit, but generates heaps of output (entering line for each single file) and I think it breaks when using "make -j".

Steffen


CMakeLists.txt excerpt just to illustrate the idea:

        foreach(_BT_LANG "C" "CXX")
            file(WRITE "${CMAKE_BINARY_DIR}/ccache-${_BT_LANG}" ""
                "#!/bin/sh\n"
                "# Xcode generator doesn't include the compiler as the
first arg:\n"
                "[ \"$1\" = \"${CMAKE_${_BT_LANG}_COMPILER}\" ] && shift\n"
                "\n"
                "# \"Convert\" cmake variable to an env to allow
shared caches\n"
                "export CCACHE_BASEDIR=${_BT_CCACHE_BASEDIR}\n"
                "export CCACHE_SLOPPINESS=file_macro,time_macros\n"
                "# export CCACHE_NODIRECT=set\n"
                "export CCACHE_CPP2=set\n"
                "[ \"\$VERBOSE\" ] && echo \"make[\${MAKELEVEL:-1}]:
Entering directory '$(pwd)'\"\n"
                "\"${_BT_${_BT_LANG}_LAUNCHER}\"
\"${CMAKE_${_BT_LANG}_COMPILER}\" \"$@\"\n"
                "[ \"\$VERBOSE\" ] && echo \"make[\${MAKELEVEL:-1}]:
Leaving directory '$(pwd)'\"\n"
                )
            execute_process(COMMAND chmod a+rx
"${CMAKE_BINARY_DIR}/ccache-${_BT_LANG}")
        endforeach()
        ...
                CHECK_CXX_COMPILER_FLAG(-fdiagnostics-color=auto
CXX_FLAG_DIAGNOSTIC_COLOR_AUTO)
                if (${CXX_FLAG_DIAGNOSTIC_COLOR_AUTO})
                  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fdiagnostics-color=auto")
                  set(CMAKE_CXX_FLAGS  "${CMAKE_CXX_FLAGS}
-fdiagnostics-color=auto")
                endif()
            endif()
            set(CMAKE_C_COMPILER_LAUNCHER   "${CMAKE_BINARY_DIR}/ccache-C")
            set(CMAKE_CXX_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/ccache-CXX")

(Instead of VERBOSE we better check MAKEFLAGS for “w”)

As far as I am aware, you are correct the design goal of CCACHE_BASEDIR is to solve this problem.

A couple of things:

  1. You can try the ninja generator it has a consistent build directory for all invocations
  2. You might not be getting cache hits because the -g or __FILE__ macros are still being used in the hash computation ( see https://ccache.dev/manual/latest.html#_compiling_in_different_directories )
  3. IDE’s shouldn’t be parsing the makefile output. They should be using CMake file-api or a compile_commands.json. Both of those shouldn’t be effected by ccache.

Thank you for your reply!

You are right, exactly.
I think this is what CCACHE_BASEDIR shall solve. I made a minimal application (just using our proprietary logging library and its dependencies) and it rebuilds in a different source directory with 100% cache hits, so I hope I’m on right way.

But how else should the IDE know which compiler error happened where? Do you think the IDE shall mimic all the make handling and run everything on its own (instead of invoking cmake --build or alike)?

I think in practice IDEs exec cmake/make and parse its output (Vim, QTCreator, Clion).

I think it is wrong that cmake uses cd in a way that it breaks make -w, but I understand that due to usage of absolute path it normally does not matter,
which leaves the question: how to effectively use ccache with cmake.

Is no one using ccache shared cache with cmake?
Does this mean that no one uses it because it is not usable? Would be a pity, wouldn’t it?

Yes we do.
We solved the basedir issue by (volume) mounting the shared source tree in the exact same location,
say: /mount/path/to/src

  • for local dev source tree is checked out at the appropriate location
  • for in-docker dev source is mounted using docker volumes from the docker host (our nominal usage scenario)

In this way all sources always appear with the very same absolute path.
In our scenario the cache directory itself is shared among developers (and CI) which NFS mounts the shared ccache dir.

This works fine, but the shared ccache dir should have fast access so NFS mount is in our LAN.

Sorry for my late reply, seems I missed the notification mail.

Thanks for your help!

I don’t understand how this is working. If I understand correctly, the path would be the same for every user on the system. Do you all work in the same source tree? Or does everyone has their own build server? Here everyone has its own git clone working directories (typically multiple) and the big servers are shared.

Could you please give me an additional hint?