Unable to link CUDA device code with MPICH implementation

Hello, i am struggling for compiling a code that requires both CUDA and MPI (MPICH implementation). In particular, when cmake is linking CUDA device code i get the following errors:

gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc: error: unrecognized command line option ‘-rpath’
gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?

These errors are caused by the following flag passed on the nvcc command: -Xcompiler=-Wl,-rpath,-Wl,/opt/modules/binary/mpich/3.3.2/lib,-Wl,--enable-new-dtags . To work it must be either wrapped in quotes or used with Xlinker instead of Xcompiler. However, i believe that the actual culprit is MPICH, because when i ask mpicxx for compile and linking flags i get the following output:

$ mpicc -link_info
gcc -I/opt/modules/binary/mpich/3.3.2/include -L/opt/modules/binary/mpich/3.3.2/lib -Wl,-rpath -Wl,/opt/modules/binary/mpich/3.3.2/lib -Wl,–enable-new-dtags -lmpi
$ mpicc -compile_info
gcc -I/opt/modules/binary/mpich/3.3.2/include -L/opt/modules/binary/mpich/3.3.2/lib -Wl,-rpath -Wl,/opt/modules/binary/mpich/3.3.2/lib -Wl,–enable-new-dtags -lmpi

if i guess correctly what happens is that cmake correctly forward all flags that mpich provides as compiler flags to the compiler, however this cause the error. I solved the issue by setting the following CMake policy (which ignores all the link flags as far as i understood) :

cmake_policy(SET CMP0105 OLD)

However, i don’t really like this solution because eventually it will become obsolete. Does someone have a better solution? I also tried something similar to the one proposed here: https://gitlab.kitware.com/cmake/cmake/-/issues/18897 ; but it seems that i can’t use generators with INTERFACE_COMPILE_OPTIONS .

I used CMake 3.19.6 and 3.20.0 (compiled from sources), nvcc 11.2.152, and MPICH 3.3.2.

@marc.chevrier Seems there might be a problem in CMP0105? I’m not very familiar with this area of CMake personally.

I am not sure because CMP0105 is for link options, not compile options.

@Neromaglio I am surprised by your assertion INTERFACE_COMPILE_OPTIONS does not support generator expressions because this property supports generator expressions.
Moreover, can you publish a snippet, including commands used to set compile and link options, showing the problem?

Thanks for the answers. I am attaching to this reply a dummy application to trigger the behavior. But I would like to stress the fact that I don’t think that the problem lies in CMake, but in how MPICH suggests compiler and linking flags that trick CMake. For this reason, I was willing to customize the FindMPI.cmake file to hide those flags from CUDA (since I don’t use MPI from CUDA sources). So I tried to wrap them using the HOST_LINK or the COMPILE_LANGUAGE generators, but CMake complained about them. That’s why I assumed that generators were not allowed, but maybe I was using them wrong.

Regarding the example, here ( https://polimi365-my.sharepoint.com/:u:/g/personal/10278244_polimi_it/EQwelUWbnZRCgUB_bJGwZJgBL8fPgXKkNRdE7VbBLbotOw?e=xtZnKK ) you can download the whole example project. I will report here the CMakeLists.txt file and the output of configuration and building:

CMake file:

cmake_minimum_required(VERSION 3.19 FATAL_ERROR)
project(demo VERSION 1.0)

enable_language(CXX)
#cmake_policy(SET CMP0105 OLD)
enable_language(CUDA)

set(MPI_CXX_SKIP_MPICXX)
find_package(MPI REQUIRED C)

set(cxx_header_files
“${CMAKE_CURRENT_SOURCE_DIR}/src/kernel.hpp”
)
set(cxx_source_files
“${CMAKE_CURRENT_SOURCE_DIR}/src/main.cpp”
)
set(cuda_source_files
“${CMAKE_CURRENT_SOURCE_DIR}/src/kernel.cu”
)
set(source_files
${cxx_source_files}
${cuda_source_files}
${cxx_header_files}
)

add_executable(demo)
target_include_directories(demo PRIVATE “${CMAKE_CURRENT_SOURCE_DIR}/src”)
target_sources(demo PRIVATE ${source_files})
target_link_libraries(demo PRIVATE MPI::MPI_C)
set_target_properties(demo
PROPERTIES
CXX_STANDARD 14
CXX_STANDARD_REQUIRED ON
CXX_EXTENSIONS OFF
CUDA_STANDARD 14
CUDA_STANDARD_REQUIRED ON
CUDA_EXTENSIONS OFF
CUDA_SEPARABLE_COMPILATION ON
)

Project configuration:

$ cmake -DCMAKE_CUDA_ARCHITECTURES=70 …
– The C compiler identification is GNU 9.3.0
– The CXX compiler identification is GNU 9.3.0
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working C compiler: /usr/bin/cc - skipped
– Detecting C compile features
– Detecting C compile features - done
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Check for working CXX compiler: /usr/bin/c++ - skipped
– Detecting CXX compile features
– Detecting CXX compile features - done
– The CUDA compiler identification is NVIDIA 10.1.243
– Detecting CUDA compiler ABI info
– Detecting CUDA compiler ABI info - done
– Check for working CUDA compiler: /usr/bin/nvcc - skipped
– Detecting CUDA compile features
– Detecting CUDA compile features - done
– Found MPI_C: /opt/mpich/3.3.2_native/lib/libmpi.so (found version “3.1”)
– Found MPI: TRUE (found version “3.1”) found components: C
– Configuring done
– Generating done
– Build files have been written to: /home/dgadioli/projects/demo/build

Project build:

$ make VERBOSE=1
/opt/cmake/cmake-3.19.7_gcc_native/bin/cmake -S/home/dgadioli/projects/demo -B/home/dgadioli/projects/demo/build --check-build-system CMakeFiles/Makefile.cmake 0
/opt/cmake/cmake-3.19.7_gcc_native/bin/cmake -E cmake_progress_start /home/dgadioli/projects/demo/build/CMakeFiles /home/dgadioli/projects/demo/build//CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory ‘/home/dgadioli/projects/demo/build’
make -f CMakeFiles/demo.dir/build.make CMakeFiles/demo.dir/depend
make[2]: Entering directory ‘/home/dgadioli/projects/demo/build’
cd /home/dgadioli/projects/demo/build && /opt/cmake/cmake-3.19.7_gcc_native/bin/cmake -E cmake_depends “Unix Makefiles” /home/dgadioli/projects/demo /home/dgadioli/projects/demo /home/dgadioli/projects/demo/build /home/dgadioli/projects/demo/build /home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/DependInfo.cmake --color=
Dependee “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/DependInfo.cmake” is newer than depender “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/depend.internal”.
Dependee “/home/dgadioli/projects/demo/build/CMakeFiles/CMakeDirectoryInformation.cmake” is newer than depender “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/depend.internal”.
Scanning dependencies of target demo
make[2]: Leaving directory ‘/home/dgadioli/projects/demo/build’
make -f CMakeFiles/demo.dir/build.make CMakeFiles/demo.dir/build
make[2]: Entering directory ‘/home/dgadioli/projects/demo/build’
[ 25%] Building CXX object CMakeFiles/demo.dir/src/main.cpp.o
/usr/bin/c++ -I/home/dgadioli/projects/demo/src -std=c++14 -o CMakeFiles/demo.dir/src/main.cpp.o -c /home/dgadioli/projects/demo/src/main.cpp
[ 50%] Building CUDA object CMakeFiles/demo.dir/src/kernel.cu.o
/usr/bin/nvcc -I/home/dgadioli/projects/demo/src --generate-code=arch=compute_70,code=[compute_70,sm_70] -std=c++14 -x cu -dc /home/dgadioli/projects/demo/src/kernel.cu -o CMakeFiles/demo.dir/src/kernel.cu.o
[ 75%] Linking CUDA device code CMakeFiles/demo.dir/cmake_device_link.o
/opt/cmake/cmake-3.19.7_gcc_native/bin/cmake -E cmake_link_script CMakeFiles/demo.dir/dlink.txt --verbose=1
/usr/bin/nvcc --generate-code=arch=compute_70,code=[compute_70,sm_70] -Xcompiler=-Wl,-rpath,-Wl,/opt/mpich/3.3.2_native/lib,-Wl,–enable-new-dtags -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink CMakeFiles/demo.dir/src/main.cpp.o CMakeFiles/demo.dir/src/kernel.cu.o -o CMakeFiles/demo.dir/cmake_device_link.o -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/gcc/x86_64-linux-gnu/8 -lcudadevrt -lcudart_static -lrt -lpthread -ldl
gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc-8: error: unrecognized command line option ‘-rpath’
gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
make[2]: *** [CMakeFiles/demo.dir/build.make:119: CMakeFiles/demo.dir/cmake_device_link.o] Error 1
make[2]: Leaving directory ‘/home/dgadioli/projects/demo/build’
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/demo.dir/all] Error 2
make[1]: Leaving directory ‘/home/dgadioli/projects/demo/build’
make: *** [Makefile:103: all] Error 2

HOST_LINK is only usable in LINK_OPTIONS and INTERFACE_LINK_OPTIONS properties.

From what is specified for link step, the option -Xcompiler=-Wl,-rpath,-Wl,/opt/mpich/3.3.2_native/lib,-Wl,–enable-new-dtags' is erroneous because the following is delivered to underneath linker (i.e. gcc-8in your case):-Wl -rpath -Wl /opt/mpich/3.3.2_native/lib -Wl –enable-new-dtagsbut the expected syntax forgcc-8is-Wl,-rpath,/opt/mpich/3.3.2_native/lib -Wl,–enable-new-dtags. This is why gcc-8complains about option-Wlor-rpath`, etc…

So, the problem is clearly in the FindMPI module: the handling of flags using -Wl must be re-worked…

Ok, thank you. Should i open a bug report? I am kinda cautious because I think that those flags are not supposed to appear in the compile information provided by MPICH in the first place, since they are linking flags. That’s why I was looking for a workaround.

The root of the problem is effectively on how link options are expressed.
With CMP0105 at NEW, link options are propagated to the device link step with a wrong syntax. Here there are two possibilities: Wrapping these link options in $<HOST_LINK:...> genex to avoid to propagate them to the device link step or fix the syntax (May be using -Xlinker) if propagating these link options to device link step is meaningful.

Now, using $<HOST_LINK:...> require some precautions regarding arguments: comma must be transformed in $<COMMA> genex to avoid wrong expansion.

After further investigation, the real problem is not about how link options are specified by MPI module but rather how options are forwarded to the underlying compiler (in your case gcc).
To propagate the options, CMake use the prefix -Xcompiler. Unfortunately, the options contains options with prefix -Wl and the option -Xcompiler cannot be used in this case.
To handle -Wl prefix, the option -Xlinker must be used instead (i.e. -Wl,opt1,opt2 must be transformed in -Xlinker=opt,opt2). To support this, It is required to re-work how CMake generates link options for CUDA compiler.

Now, for MPI module, I think the correct approach is to update the module to wrap link options inside $<HOST_LINK:...> genex to avoid propagating them to device link step. But I am not an expert of CUDA environments.

So you can create an issue for MPI module describing the problem and add reference to this discussion. CUDA experts will be able to comment and validate (or not) my fix suggestion.

I will manage the creation of another issue for device link handling.

@Neromaglio No need to create an issue. Someone already did: see #21887.