# Unable to link CUDA device code with MPICH implementation
**URL:** https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006
**Category:** Usage
**Tags:** os:linux
**Created:** [March 24, 2021, 3:31pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006 "2021-03-24T15:31:28Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Neromaglio](https://discourse.cmake.org/u/Neromaglio)
#### Post date: [March 24, 2021, 3:31pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/1 "2021-03-24T15:31:28Z")
Hello, i am struggling for compiling a code that requires both CUDA and MPI (MPICH implementation). In particular, when cmake is linking CUDA device code i get the following errors:
```
gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc: error: unrecognized command line option ‘-rpath’
gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
gcc: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
```
These errors are caused by the following flag passed on the nvcc command: `-Xcompiler=-Wl,-rpath,-Wl,/opt/modules/binary/mpich/3.3.2/lib,-Wl,--enable-new-dtags` . To work it must be either wrapped in quotes or used with Xlinker instead of Xcompiler. However, i believe that the actual culprit is MPICH, because when i ask mpicxx for compile and linking flags i get the following output:
> $ mpicc -link\_info
> gcc -I/opt/modules/binary/mpich/3.3.2/include -L/opt/modules/binary/mpich/3.3.2/lib -Wl,-rpath -Wl,/opt/modules/binary/mpich/3.3.2/lib -Wl,–enable-new-dtags -lmpi
> $ mpicc -compile\_info
> gcc -I/opt/modules/binary/mpich/3.3.2/include -L/opt/modules/binary/mpich/3.3.2/lib -Wl,-rpath -Wl,/opt/modules/binary/mpich/3.3.2/lib -Wl,–enable-new-dtags -lmpi
if i guess correctly what happens is that cmake correctly forward all flags that mpich provides as compiler flags to the compiler, however this cause the error. I solved the issue by setting the following CMake policy (which ignores all the link flags as far as i understood) :
> cmake\_policy(SET CMP0105 OLD)
However, i don’t really like this solution because eventually it will become obsolete. Does someone have a better solution? I also tried something similar to the one proposed here: [https://gitlab.kitware.com/cmake/cmake/-/issues/18897](https://gitlab.kitware.com/cmake/cmake/-/issues/18897) ; but it seems that i can’t use generators with INTERFACE\_COMPILE\_OPTIONS .
I used CMake 3.19.6 and 3.20.0 (compiled from sources), nvcc 11.2.152, and MPICH 3.3.2.
---
### Author:  [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [March 25, 2021, 6:20pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/2 "2021-03-25T18:20:07Z")
@marc.chevrier Seems there might be a problem in CMP0105? I’m not very familiar with this area of CMake personally.
---
### Author:  [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)
#### Post date: [March 26, 2021, 12:56pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/3 "2021-03-26T12:56:35Z")
I am not sure because `CMP0105` is for link options, not compile options.
@Neromaglio I am surprised by your assertion `INTERFACE_COMPILE_OPTIONS does not support generator expressions` because this property supports generator expressions.
Moreover, can you publish a snippet, including commands used to set compile **and** link options, showing the problem?
---
### Author:  [@Neromaglio](https://discourse.cmake.org/u/Neromaglio)
#### Post date: [March 26, 2021, 3:55pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/4 "2021-03-26T15:55:02Z")
Thanks for the answers. I am attaching to this reply a dummy application to trigger the behavior. But I would like to stress the fact that I don’t think that the problem lies in CMake, but in how MPICH suggests compiler and linking flags that trick CMake. For this reason, I was willing to customize the FindMPI.cmake file to hide those flags from CUDA (since I don’t use MPI from CUDA sources). So I tried to wrap them using the HOST\_LINK or the COMPILE\_LANGUAGE generators, but CMake complained about them. That’s why I assumed that generators were not allowed, but maybe I was using them wrong.
Regarding the example, here ( [Sign in to your account](https://polimi365-my.sharepoint.com/:u:/g/personal/10278244_polimi_it/EQwelUWbnZRCgUB_bJGwZJgBL8fPgXKkNRdE7VbBLbotOw?e=xtZnKK) ) you can download the whole example project. I will report here the CMakeLists.txt file and the output of configuration and building:
CMake file:
> cmake\_minimum\_required(VERSION 3.19 FATAL\_ERROR)
> project(demo VERSION 1.0)
>
> enable\_language(CXX)
> #cmake\_policy(SET CMP0105 OLD)
> enable\_language(CUDA)
>
> set(MPI\_CXX\_SKIP\_MPICXX)
> find\_package(MPI REQUIRED C)
>
> set(cxx\_header\_files
> “${CMAKE\_CURRENT\_SOURCE\_DIR}/src/kernel.hpp”
> )
> set(cxx\_source\_files
> “${CMAKE\_CURRENT\_SOURCE\_DIR}/src/main.cpp”
> )
> set(cuda\_source\_files
> “${CMAKE\_CURRENT\_SOURCE\_DIR}/src/kernel.cu”
> )
> set(source\_files
> ${cxx\_source\_files}
> ${cuda\_source\_files}
> ${cxx\_header\_files}
> )
>
> add\_executable(demo)
> target\_include\_directories(demo PRIVATE “${CMAKE\_CURRENT\_SOURCE\_DIR}/src”)
> target\_sources(demo PRIVATE ${source\_files})
> target\_link\_libraries(demo PRIVATE MPI::MPI\_C)
> set\_target\_properties(demo
> PROPERTIES
> CXX\_STANDARD 14
> CXX\_STANDARD\_REQUIRED ON
> CXX\_EXTENSIONS OFF
> CUDA\_STANDARD 14
> CUDA\_STANDARD\_REQUIRED ON
> CUDA\_EXTENSIONS OFF
> CUDA\_SEPARABLE\_COMPILATION ON
> )
Project configuration:
> $ cmake -DCMAKE\_CUDA\_ARCHITECTURES=70 …
> – The C compiler identification is GNU 9.3.0
> – The CXX compiler identification is GNU 9.3.0
> – Detecting C compiler ABI info
> – Detecting C compiler ABI info - done
> – Check for working C compiler: /usr/bin/cc - skipped
> – Detecting C compile features
> – Detecting C compile features - done
> – Detecting CXX compiler ABI info
> – Detecting CXX compiler ABI info - done
> – Check for working CXX compiler: /usr/bin/c++ - skipped
> – Detecting CXX compile features
> – Detecting CXX compile features - done
> – The CUDA compiler identification is NVIDIA 10.1.243
> – Detecting CUDA compiler ABI info
> – Detecting CUDA compiler ABI info - done
> – Check for working CUDA compiler: /usr/bin/nvcc - skipped
> – Detecting CUDA compile features
> – Detecting CUDA compile features - done
> – Found MPI\_C: /opt/mpich/3.3.2\_native/lib/libmpi.so (found version “3.1”)
> – Found MPI: TRUE (found version “3.1”) found components: C
> – Configuring done
> – Generating done
> – Build files have been written to: /home/dgadioli/projects/demo/build
Project build:
> $ make VERBOSE=1
> /opt/cmake/cmake-3.19.7\_gcc\_native/bin/cmake -S/home/dgadioli/projects/demo -B/home/dgadioli/projects/demo/build --check-build-system CMakeFiles/Makefile.cmake 0
> /opt/cmake/cmake-3.19.7\_gcc\_native/bin/cmake -E cmake\_progress\_start /home/dgadioli/projects/demo/build/CMakeFiles /home/dgadioli/projects/demo/build//CMakeFiles/progress.marks
> make -f CMakeFiles/Makefile2 all
> make[1]: Entering directory ‘/home/dgadioli/projects/demo/build’
> make -f CMakeFiles/demo.dir/build.make CMakeFiles/demo.dir/depend
> make[2]: Entering directory ‘/home/dgadioli/projects/demo/build’
> cd /home/dgadioli/projects/demo/build && /opt/cmake/cmake-3.19.7\_gcc\_native/bin/cmake -E cmake\_depends “Unix Makefiles” /home/dgadioli/projects/demo /home/dgadioli/projects/demo /home/dgadioli/projects/demo/build /home/dgadioli/projects/demo/build /home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/DependInfo.cmake --color=
> Dependee “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/DependInfo.cmake” is newer than depender “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/depend.internal”.
> Dependee “/home/dgadioli/projects/demo/build/CMakeFiles/CMakeDirectoryInformation.cmake” is newer than depender “/home/dgadioli/projects/demo/build/CMakeFiles/demo.dir/depend.internal”.
> Scanning dependencies of target demo
> make[2]: Leaving directory ‘/home/dgadioli/projects/demo/build’
> make -f CMakeFiles/demo.dir/build.make CMakeFiles/demo.dir/build
> make[2]: Entering directory ‘/home/dgadioli/projects/demo/build’
> [25%] Building CXX object CMakeFiles/demo.dir/src/main.cpp.o
> /usr/bin/c++ -I/home/dgadioli/projects/demo/src -std=c++14 -o CMakeFiles/demo.dir/src/main.cpp.o -c /home/dgadioli/projects/demo/src/main.cpp
> [50%] Building CUDA object CMakeFiles/demo.dir/src/kernel.cu.o
> /usr/bin/nvcc -I/home/dgadioli/projects/demo/src --generate-code=arch=compute\_70,code=[compute\_70,sm\_70] -std=c++14 -x cu -dc /home/dgadioli/projects/demo/src/kernel.cu -o CMakeFiles/demo.dir/src/kernel.cu.o
> [75%] Linking CUDA device code CMakeFiles/demo.dir/cmake\_device\_link.o
> /opt/cmake/cmake-3.19.7\_gcc\_native/bin/cmake -E cmake\_link\_script CMakeFiles/demo.dir/dlink.txt --verbose=1
> /usr/bin/nvcc --generate-code=arch=compute\_70,code=[compute\_70,sm\_70] -Xcompiler=-Wl,-rpath,-Wl,/opt/mpich/3.3.2\_native/lib,-Wl,–enable-new-dtags -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink CMakeFiles/demo.dir/src/main.cpp.o CMakeFiles/demo.dir/src/kernel.cu.o -o CMakeFiles/demo.dir/cmake\_device\_link.o -L/usr/lib/x86\_64-linux-gnu/stubs -L/usr/lib/gcc/x86\_64-linux-gnu/8 -lcudadevrt -lcudart\_static -lrt -lpthread -ldl
> gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
> gcc-8: error: unrecognized command line option ‘-rpath’
> gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
> gcc-8: error: unrecognized command line option ‘-Wl’; did you mean ‘-W’?
> make[2]: \*\*\* [CMakeFiles/demo.dir/build.make:119: CMakeFiles/demo.dir/cmake\_device\_link.o] Error 1
> make[2]: Leaving directory ‘/home/dgadioli/projects/demo/build’
> make[1]: \*\*\* [CMakeFiles/Makefile2:95: CMakeFiles/demo.dir/all] Error 2
> make[1]: Leaving directory ‘/home/dgadioli/projects/demo/build’
> make: \*\*\* [Makefile:103: all] Error 2
---
### Author:  [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)
#### Post date: [March 26, 2021, 4:46pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/5 "2021-03-26T16:46:55Z")
`HOST_LINK` is only usable in `LINK_OPTIONS` and `INTERFACE_LINK_OPTIONS` properties.
From what is specified for link step, the option `-Xcompiler=-Wl,-rpath,-Wl,/opt/mpich/3.3.2_native/lib,-Wl,–enable-new-dtags' is erroneous because the following is delivered to underneath linker (i.e. `gcc-8`in your case):`-Wl -rpath -Wl /opt/mpich/3.3.2\_native/lib -Wl –enable-new-dtags`but the expected syntax for`gcc-8`is`-Wl,-rpath,/opt/mpich/3.3.2\_native/lib -Wl,–enable-new-dtags`. This is why `gcc-8`complains about option`-Wl`or`-rpath`, etc…
So, the problem is clearly in the `FindMPI` module: the handling of flags using `-Wl` must be re-worked…
---
### Author:  [@Neromaglio](https://discourse.cmake.org/u/Neromaglio)
#### Post date: [March 26, 2021, 5:08pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/6 "2021-03-26T17:08:30Z")
Ok, thank you. Should i open a bug report? I am kinda cautious because I think that those flags are not supposed to appear in the compile information provided by MPICH in the first place, since they are linking flags. That’s why I was looking for a workaround.
---
### Author:  [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)
#### Post date: [March 27, 2021, 11:19am UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/7 "2021-03-27T11:19:42Z")
The root of the problem is effectively on how link options are expressed.
With `CMP0105` at `NEW`, link options are propagated to the device link step with a wrong syntax. Here there are two possibilities: Wrapping these link options in `$` genex to avoid to propagate them to the device link step or fix the syntax (May be using `-Xlinker`) **if propagating these link options to device link step is meaningful**.
Now, using `$` require some precautions regarding arguments: comma must be transformed in `$` genex to avoid wrong expansion.
---
### Author:  [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)
#### Post date: [March 28, 2021, 10:09am UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/8 "2021-03-28T10:09:54Z")
After further investigation, the real problem is not about how link options are specified by `MPI` module but rather how options are forwarded to the underlying compiler (in your case `gcc`).
To propagate the options, `CMake` use the prefix `-Xcompiler`. Unfortunately, the options contains options with prefix `-Wl` and the option `-Xcompiler` cannot be used in this case.
To handle `-Wl` prefix, the option `-Xlinker` must be used instead (i.e. `-Wl,opt1,opt2` must be transformed in `-Xlinker=opt,opt2`). To support this, It is required to re-work how `CMake` generates link options for `CUDA` compiler.
Now, for `MPI` module, I think the correct approach is to update the module to wrap link options inside `$` genex to avoid propagating them to device link step. But I am not an expert of `CUDA` environments.
So you can create an [issue](https://gitlab.kitware.com/cmake/cmake/-/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=) for `MPI` module describing the problem and add reference to this discussion. `CUDA` experts will be able to comment and validate (or not) my fix suggestion.
I will manage the creation of another issue for device link handling.
---
### Author:  [@marc.chevrier](https://discourse.cmake.org/u/marc.chevrier)
#### Post date: [March 28, 2021, 1:42pm UTC](https://discourse.cmake.org/t/unable-to-link-cuda-device-code-with-mpich-implementation/3006/9 "2021-03-28T13:42:32Z")
@Neromaglio No need to create an issue. Someone already did: see [#21887](https://gitlab.kitware.com/cmake/cmake/-/issues/21887).