Correct use of CMAKE_CUDA_ARCHITECTURES

Code
1

I was looking for ways to properly target different compute capabilities of cuda devices and found a couple of new policies for 3.18. So I tried (simplified):

cmake_minimum_required(VERSION 3.17 FATAL_ERROR)

cmake_policy(SET CMP0104 NEW)
cmake_policy(SET CMP0105 NEW)
...
add_library(hello SHARED hello.cpp hello.h hello.cu)
set_property(TARGET hello PROPERTY CUDA_ARCHITECTURES 52 61 75)

During configuration I have tried to set CMAKE_CUDA_ARCHITECTURES to 52 61 75 in the GUI, but the compiler test fails with

      Compiling CUDA source file main.cu...

      C:\MyProject\build\vs2019-x64\CMakeFiles\CMakeTmp>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\nvcc.exe"  --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.27.28826\bin\HostX64\x64" -x cu   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static 61 75,code=[compute_52 61 75,sm_52 61 75] -Xcompiler="/EHsc -Zi -Ob0" -g   -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /FdcmTC_e9572.dir\Debug\vc142.pdb /FS /Zi /RTC1 /MDd /GR" -o cmTC_e9572.dir\Debug\main.obj "C:\MyProject\build\vs2019-x64\CMakeFiles\CMakeTmp\main.cu" 

      nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified

Note the strange flag --compile -cudart static 61 75,code=[compute_52 61 75,sm_52 61 75]

I installed CMake 3.17.20200520-g81e8f62 on Windows, I’m using Visual Studio 10.19 (16.7.0 Preview 1.0) with NVidia’s SDK 10.2.

2

For a correct behavior, you need on-going version 3.18 but requested version3.17.

Try following syntax: cmake_minimum_required(VERSION 3.17...3.18) to get expected behavior regarding new features introduced for future v3.18.

3

I could not make this work until I switched to CMake 3.18 (rc1).
I wasn’t sure that my VS/CUDA setup wasn’t broken until the upgrade of CMake solved the problem.

But I’m experiencing a strange bug now. I’m using

if(NOT DEFINED ${CMAKE_CUDA_ARCHITECTURES})
    set(CMAKE_CUDA_ARCHITECTURES 52 61 75)
endif()
message(STATUS "CUDA architectures set to ${CMAKE_CUDA_ARCHITECTURES}")

which prints out

CUDA architectures set to 52;61;75

and the compile flag seem to be correct. However the cmake-gui still shows CMAKE_CUDA_ARCHITECTURES being set to 30.

4

cmake-gui display variables defined in the cache.

The command set(CMAKE_CUDA_ARCHITECTURES 52 61 75) defines standard variable which hide the cache variable but do not overwrite it. If you want to overwrite the cache variable, you have to edit it with cmake-gui or using the following syntax: set(<variable> <value>... CACHE <type> <docstring> FORCE), see set command.

5

Thank you. This only solves half of the problem though.

Using

set(CMAKE_CUDA_ARCHITECTURES 52 60 61 75 CACHE STRING "CUDA architectures" FORCE)

works perfectly well to set some reasonable defaults. However it sadly prevents anyone from overriding these values.

For CMAKE_INSTALL_PREFIX I’m using the following trick:

if(CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT)
    set(CMAKE_INSTALL_PREFIX "/some/path" CACHE PATH "Default location for installing the library" FORCE)
endif(CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT)

but I cannot figure out how to do something similar for CUDA architectures. All I want is basically just replace the default defaults of 30 to something more reasonable for our project, but still give the user building the binaries some freedom.

6

Not at all. When, in a CMakeLists.txt, the variable is set (without CACHE specification), this variable takes precedence over the cache definition.

For example

# First definition in the cache
set(CMAKE_CUDA_ARCHITECTURES 52 60 61 75 CACHE STRING "CUDA architectures")

#  ... after in some CMakeLists.txt
set (CMAKE_CUDA_ARCHITECTURES 52 60 70 90)
message ("CUDA ARCHS: ${CMAKE_CUDA_ARCHITECTURES}")

You get value CUDA ARCHS: 52;60;70;90 displayed, not the value defined in the cache.

The idea behind the cache is to ensure some defaults are available each time cmake is relaunched without preventing the user to override these defaults.

So the cache value is defined on the command line or with ccmake or ccmake-gui and after, in the CMakeLists.txt files, it is possible to override this default easily by just setting the variable (set command without CACHE option) without changing the cache value.

7

Thank you for the explanation, it clarified some aspects of how CMake handles cache, so I can better understand what is going on under the hood, but I still don’t quite get it how to achieve what I’m striving for.

For simplicity let’s assume that we only have a trivial hello-world library with a single CMakeLists.txt, so there is no need for different CMakeLists.txt inside the project to override each other. (Still, I don’t want to specify this property on per-target basis; I just want to be able to specify the list of architectures at a single place and keep it consistent across the project.) I would like to support the following two scenarios:

Case 1: The user compiling the library doesn’t do anything extra (no extra options are specified). I would want to compile support for architectures “52 60 61 75” in that case.

Case 2: The user compiling the library runs cmake-gui, selects Configure. The value of CMAKE_CUDA_ARCHITECTURES should read 52;60;61;75 (not 30). The user changes that value to 70, clicks Configure, then Generate. The option 70 stays (it should not be automatically changed back to 52;60;61;75) and the generated project has the correct flags just for architecture 70.

The code

set(CMAKE_CUDA_ARCHITECTURES 52 60 61 75)

is not really helpful. It does work the way you describe it, it does have the desired effect of enforcing “my favourite” list of architectures, but first of all it lies to the user about the architectures being used (the GUI claims that the architecture 30 would be used) which is not too helpful, and even more importantly it prevents the user from having any effect on the set of architectures being used for the actual compilation. (It does keep the user selection shown, but that selection is ignored.)

By using

set(CMAKE_CUDA_ARCHITECTURES 52 60 61 75 CACHE STRING "CUDA architectures" FORCE)

on the other hand the cmake-gui correctly reports the set of selected architectures to the user. But there doesn’t seem to be any way to change that selection. Whenever the user changes the value and clicks Configure or Generate, the values are reset back to 52;60;61;75.

Maybe something along the lines of the following code could have worked (or at least something similar works for setting the default prefix):

if(CMAKE_CUDA_ARCHITECTURES_INITIALIZED_TO_DEFAULT)
    set(CMAKE_CUDA_ARCHITECTURES 52 60 61 75 CACHE STRING "CUDA architectures" FORCE)
endif(CMAKE_CUDA_ARCHITECTURES_INITIALIZED_TO_DEFAULT)

except that no such function (CMAKE_CUDA_ARCHITECTURES_INITIALIZED_TO_DEFAULT) exists yet, and introducing one probably doesn’t scale either.

8

I don’t really see the value of hardcoding binary formats into a build script, since this list is likely a user preference depending on what machines they wish to target with their build. I tried supplying CMAKE_CUDA_ARCHITECTURES from the command-line, passing two parameters, but it never works. With the value that’s being passed being the literal 52 70 or 52;70 neither work and results in nvcc compiler check failures.

[cmake] -- Check for working CUDA compiler: /usr/local/cuda-10.2/bin/nvcc - broken
[cmake] CMake Error at /snap/cmake/619/share/cmake-3.18/Modules/CMakeTestCUDACompiler.cmake:52 (message):
[cmake]   The CUDA compiler
[cmake] 
[cmake]     "/usr/local/cuda-10.2/bin/nvcc"
[cmake] 
[cmake]   is not able to compile a simple test program.
[cmake] 
[cmake]   It fails with the following output:
[cmake] 
[cmake]     Change Dir: /home/mate/Source/Repos/gromacs/src/benchmark_amd/.vscode/build/CMakeFiles/CMakeTmp
[cmake]     
[cmake]     Run Build Command(s):/home/mate/.local/bin/ninja cmTC_dc8b7 && [1/2] Building CUDA object CMakeFiles/cmTC_dc8b7.dir/main.cu.o
[cmake]     FAILED: CMakeFiles/cmTC_dc8b7.dir/main.cu.o 
[cmake]     /usr/local/cuda-10.2/bin/nvcc -forward-unknown-to-host-compiler   --generate-code=arch=compute_52;70,code=[compute_52;70,sm_52;70] -MD -MT CMakeFiles/cmTC_dc8b7.dir/main.cu.o -MF CMakeFiles/cmTC_dc8b7.dir/main.cu.o.d -x cu -c main.cu -o CMakeFiles/cmTC_dc8b7.dir/main.cu.o
[cmake]     nvcc fatal   : Option '--generate-code arch=compute_52', missing code
[cmake]     /bin/sh: 1: 70,code=[compute_52: not found
[cmake]     /bin/sh: 1: 70,sm_52: not found
[cmake]     /bin/sh: 1: 70]: not found
[cmake]     ninja: build stopped: subcommand failed.

The command assembled by CMake is garbage in different ways depending on whether there’s a space or a semi-colon inside the command. The variable when taken from the command-line seems to be taken verbatim and isn’t interpreted as a CMake list.

(Note that I omit the details of how many quotes and escapes needed. I’m parametrizing the CMake invocation from VS Code via CMake Tools. I’m checking the actual command-line invocation and CMake Tools seems to correctly quote/escape/massage command-line args when containing apostrophes, backslashes, semi-colons, etc. I experience all this using cmake version 3.18.4)