Using --keep and --keep-dir with CUDA compilation commands

Hi,

I ran into an unexpected complication with our project, which I’d like to collect some ideas about. In our project, amongst other things, we build some CUDA source code. In order to do some not-completely-trivial debugging on our code, we ask nvcc to leave some intermediate files behind for us, which we would analyze with some custom scripts. (Looking for certain errors, patterns, etc.)

The way we’ve been doing this, was to add something along the lines of

set(CMAKE_CUDA_FLAGS_DEBUG "${CMAKE_CUDA_FLAGS_DEBUG} -G --keep --source-in-ptx")

to our configuration. Which worked for us so far.

But we recently added two identically named .cu files in two different subdirectories, which would be built into a single shared library. And that caused a long list of hard-to-reproduce issues in our build until I finally understood that this was all because of multiple nvcc processes trying to write to the same file on disk, and getting very confused about this. (Which would of course only cause an issue in our build if the two source files happened to be built at the same time.)

The nvcc executable also has a --keep-dir argument that would be exactly the sort of thing that we need here.

But I can just not figure out how I could tell our build system to save the intermediate files generated by --keep, into “source file specific” subdirectories. Is there some generator expression way of achieving this? Similar to how object files are being put into separate subdirectories in this setup just fine. :thinking:

Cheers,
Attila

This issue is of interest here:

https://gitlab.kitware.com/cmake/cmake/-/issues/21986

Thanks for the pointer Ben!

I tried to add the --keep-dir argument to CMAKE_CUDA_COMPILE_OBJECT, but couldn’t find an incantation yet for getting the directory name out of <OBJECT>. Is there a way of doing that? Since unfortunately the directory has to already exist when nvcc gets called, so I couldn’t just do something like --keep-dir <OBJECT>.cudakeep. :slightly_frowning_face:

Source file or object file specific generator properties could be interesting, but maybe not absolutely necessary for this particular purpose. :thinking:

Yeah, we don’t model that at the moment. You could set CMAKE_CUDA_COMPILER_LAUNCHER and do what you need there in the meantime.