# Using --keep and --keep-dir with CUDA compilation commands

**URL:** https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307
**Category:** Code
**Tags:** lang:cuda
**Created:** [October 30, 2023, 1:55pm UTC](https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307 "2023-10-30T13:55:30Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![krasznaa](https://discourse.cmake.org/user_avatar/discourse.cmake.org/krasznaa/32/893_2.png) [@krasznaa](https://discourse.cmake.org/u/krasznaa)
#### Post date: [October 30, 2023, 1:55pm UTC](https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307/1 "2023-10-30T13:55:30Z")

</div>

Hi,

I ran into an unexpected complication with our project, which I’d like to collect some ideas about. In our project, amongst other things, we build some CUDA source code. In order to do some not-completely-trivial debugging on our code, we ask `nvcc` to leave some intermediate files behind for us, which we would analyze with some custom scripts. (Looking for certain errors, patterns, etc.)

The way we’ve been doing this, was to add something along the lines of

```cmake
set(CMAKE_CUDA_FLAGS_DEBUG "${CMAKE_CUDA_FLAGS_DEBUG} -G --keep --source-in-ptx")

```

to our configuration. Which worked for us so far.

But we recently added two identically named `.cu` files in two different subdirectories, which would be built into a single shared library. And that caused a long list of hard-to-reproduce issues in our build until I finally understood that this was all because of multiple `nvcc` processes trying to write to the same file on disk, and getting very confused about this. (Which would of course only cause an issue in our build if the two source files happened to be built at the same time.)

The `nvcc` executable also has a `--keep-dir` argument that would be exactly the sort of thing that we need here.

> **[1. Introduction — NVIDIA CUDA Compiler Driver 12.3 documentation](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#keep-dir-directory-keep-dir)**
>
> The documentation for nvcc, the CUDA compiler driver.

But I can just not figure out how I could tell our build system to save the intermediate files generated by `--keep`, into “source file specific” subdirectories. Is there some generator expression way of achieving this? Similar to how object files are being put into separate subdirectories in this setup just fine. 🤔

Cheers,  
Attila

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [October 30, 2023, 2:01pm UTC](https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307/2 "2023-10-30T14:01:41Z")

</div>

This issue is of interest here:

[https://gitlab.kitware.com/cmake/cmake/-/issues/21986](https://gitlab.kitware.com/cmake/cmake/-/issues/21986)

---

<div class="post-metadata">

### Author: ![krasznaa](https://discourse.cmake.org/user_avatar/discourse.cmake.org/krasznaa/32/893_2.png) [@krasznaa](https://discourse.cmake.org/u/krasznaa)
#### Post date: [October 30, 2023, 2:21pm UTC](https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307/3 "2023-10-30T14:21:35Z")

</div>

Thanks for the pointer Ben!

I tried to add the `--keep-dir` argument to `CMAKE_CUDA_COMPILE_OBJECT`, but couldn’t find an incantation yet for getting the directory name out of `<OBJECT>`. Is there a way of doing that? Since unfortunately the directory has to already exist when `nvcc` gets called, so I couldn’t just do something like `--keep-dir <OBJECT>.cudakeep`. 🙁

Source file or object file specific generator properties could be interesting, but maybe not absolutely necessary for this particular purpose. 🤔

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [October 30, 2023, 3:19pm UTC](https://discourse.cmake.org/t/using-keep-and-keep-dir-with-cuda-compilation-commands/9307/4 "2023-10-30T15:19:21Z")

</div>

Yeah, we don’t model that at the moment. You could set [`CMAKE_CUDA_COMPILER_LAUNCHER`](https://cmake.org/cmake/help/latest/prop_tgt/LANG_LINKER_LAUNCHER.html) and do what you need there in the meantime.