Using --keep and --keep-dir with CUDA compilation commands

Attila Krasznahorkay
Mon, 30 Oct 2023 13:55:30 +0000
Code

Hi,

I ran into an unexpected complication with our project, which I’d like to collect some ideas about. In our project, amongst other things, we build some CUDA source code. In order to do some not-completely-trivial debugging on our code, we ask nvcc to leave some intermediate files behind for us, which we would analyze with some custom scripts. (Looking for certain errors, patterns, etc.)

The way we’ve been doing this, was to add something along the lines of

set(CMAKE_CUDA_FLAGS_DEBUG "${CMAKE_CUDA_FLAGS_DEBUG} -G --keep --source-in-ptx")

to our configuration. Which worked for us so far.

But we recently added two identically named .cu files in two different subdirectories, which would be built into a single shared library. And that caused a long list of hard-to-reproduce issues in our build until I finally understood that this was all because of multiple nvcc processes trying to write to the same file on disk, and getting very confused about this. (Which would of course only cause an issue in our build if the two source files happened to be built at the same time.)

The nvcc executable also has a --keep-dir argument that would be exactly the sort of thing that we need here.

docs.nvidia.com

1. Introduction — NVIDIA CUDA Compiler Driver 12.3 documentation

The documentation for nvcc, the CUDA compiler driver.

But I can just not figure out how I could tell our build system to save the intermediate files generated by --keep, into “source file specific” subdirectories. Is there some generator expression way of achieving this? Similar to how object files are being put into separate subdirectories in this setup just fine.

Cheers,
Attila

4 posts - 2 participants