Building CUDA code with a Clang host compiler which uses a non-standard GCC installation

Attila Krasznahorkay
Thu, 08 May 2025 08:58:08 +0000
Code

Hi,

The issue I bumped into just now may be a very corner case thing, but I think I found a bug…

One of the environments in which I’d like to build CUDA code is one where host code is being compiled with a version of clang, which itself was not built against the system’s default installation of GCC, but rather a custom GCC installation. I have the following set up:

CXX="/software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++"
CXXFLAGS="--gcc-toolchain=/software/gcc/13.3.0/x86_64-el9"

CUDACXX="/software/cuda/12.8.1/x86_64/bin/nvcc"
CUDAHOSTCXX="/software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++"
CUDAFLAGS="-Xcompiler --gcc-toolchain=/software/gcc/13.3.0/x86_64-el9 -allow-unsupported-compiler"

In this environment, I try to build the following simple project:

# Set up the project.
cmake_minimum_required(VERSION 3.20)
project(CMakeCUDABug VERSION 1.0.0 LANGUAGES CXX CUDA)

# Build a simple CUDA executable.
add_executable(example source.cpp source.cu)

The contents of source.cpp and source.cu are not important, the important bit is just that I link both C++ and CUDA code into a single binary.

In the environment that I described earlier, I see the following:

[bash][atspot01]:development > cmake -S cuda_bug/ -B build
-- The CXX compiler identification is IntelLLVM 2025.1.0
-- The CUDA compiler identification is NVIDIA 12.8.93 with host compiler IntelLLVM 2025.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /software/cuda/12.8.1/x86_64/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done (2.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/krasznaa/development/build
[bash][atspot01]:development > VERBOSE=1 cmake --build build/
Change Dir: '/home/krasznaa/development/build'

Run Build Command(s): /software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile
/software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -S/home/krasznaa/development/cuda_bug -B/home/krasznaa/development/build --check-build-system CMakeFiles/Makefile.cmake 0
/software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -E cmake_progress_start /home/krasznaa/development/build/CMakeFiles /home/krasznaa/development/build//CMakeFiles/progress.marks
/usr/bin/gmake  -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/home/krasznaa/development/build'
/usr/bin/gmake  -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/depend
gmake[2]: Entering directory '/home/krasznaa/development/build'
cd /home/krasznaa/development/build && /software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -E cmake_depends "Unix Makefiles" /home/krasznaa/development/cuda_bug /home/krasznaa/development/cuda_bug /home/krasznaa/development/build /home/krasznaa/development/build /home/krasznaa/development/build/CMakeFiles/example.dir/DependInfo.cmake "--color="
gmake[2]: Leaving directory '/home/krasznaa/development/build'
/usr/bin/gmake  -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/build
gmake[2]: Entering directory '/home/krasznaa/development/build'
[ 33%] Building CXX object CMakeFiles/example.dir/source.cpp.o
/software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++   --gcc-toolchain=/software/gcc/13.3.0/x86_64-el9 -MD -MT CMakeFiles/example.dir/source.cpp.o -MF CMakeFiles/example.dir/source.cpp.o.d -o CMakeFiles/example.dir/source.cpp.o -c /home/krasznaa/development/cuda_bug/source.cpp
[ 66%] Building CUDA object CMakeFiles/example.dir/source.cu.o
/software/cuda/12.8.1/x86_64/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++   -Xcompiler --gcc-toolchain=/software/gcc/13.3.0/x86_64-el9 -allow-unsupported-compiler -std=c++17 "--generate-code=arch=compute_52,code=[compute_52,sm_52]" -MD -MT CMakeFiles/example.dir/source.cu.o -MF CMakeFiles/example.dir/source.cu.o.d -x cu -c /home/krasznaa/development/cuda_bug/source.cu -o CMakeFiles/example.dir/source.cu.o
nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
[100%] Linking CXX executable example
/software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -E cmake_link_script CMakeFiles/example.dir/link.txt --verbose=1
/software/intel/oneapi-2025.1.0/compiler/2025.1/bin/compiler/clang++ --gcc-toolchain=/software/gcc/13.3.0/x86_64-el9 CMakeFiles/example.dir/source.cpp.o CMakeFiles/example.dir/source.cu.o -o example   -L/software/cuda/12.8.1/x86_64/targets/x86_64-linux/lib/stubs  -L/software/cuda/12.8.1/x86_64/targets/x86_64-linux/lib  -L/usr/lib/gcc/x86_64-redhat-linux/11  -lcudadevrt -lcudart_static -lrt -lpthread -ldl
gmake[2]: Leaving directory '/home/krasznaa/development/build'
[100%] Built target example
gmake[1]: Leaving directory '/home/krasznaa/development/build'
/software/cmake/4.0.2/x86_64-el9-gcc13-opt/bin/cmake -E cmake_progress_start /home/krasznaa/development/build/CMakeFiles 0

[bash][atspot01]:development >

The issue hides very sneakily in the final link command. The compilation commands work as they should, they set up clang++ and nvcc to behave how I’d like them to. But the final link has this element that I can’t explain:

-L/usr/lib/gcc/x86_64-redhat-linux/11

And while in this piece of demonstrator code it doesn’t cause an issue, in a real project of mine it does. It leads to a link failure, as the link tries to use an incorrect version of libstdc++.so.

I know this is a complicated build setup, but it should still be possible to make it work I hope. Did others bump into this in the past, is this a known issue?

Cheers,
Attila

1 post - 1 participant