Maintaining list of Python source files without file(GLOB_RECURSE ...)

Usage
1

Hi everyone,
I have a large project that uses CMake (3.6) & ninja for building. The code is mostly C++ but it has a few Python components as well. The build process auto-generates some Python files in ${CMAKE_BINARY_DIR}/py. I need to copy some hand-written Python files into the same directory for running unit tests and creating an sdist.

The question is about best practices for maintaining the list of these hand-written Python files. Say I have the following directory structure that needs to be copied to ${CMAKE_BINARY_DIR}/py:

a
├── b
│   ├── c
│   │   ├── CMakeLists.txt
│   │   └── __init__.py
│   ├── CMakeLists.txt
│   └── __init__.py
├── CMakeLists.txt
└── __init__.py

This is how I currently create a list of all Python source files

a.CMakeLists.txt

add_subdirectory(b)

set(A_PYTHON_SOURCES ${A_PYTHON_SOURCES}
     ${CMAKE_CURRENT_SOURCE_DIR}/__init__.py
)

add_custom_command(
    OUTPUT ${CMAKE_BINARY_DIR}/a.stamp
    DEPENDS ${A_PYTHON_SOURCES}
    COMMAND rsync -avz ...
    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_BINARY_DIR}/a.stamp
)

add_custom_target(
    build_a ALL
    DEPENDS ${CMAKE_BINARY_DIR}/a.stamp
)

b.CMakeLists.txt

add_subdirectory(c)

set(A_PYTHON_SOURCES ${A_PYTHON_SOURCES}
     ${CMAKE_CURRENT_SOURCE_DIR}/__init__.py
     PARENT_SCOPE
)

c.CMakeLists.txt

set(A_PYTHON_SOURCES ${A_PYTHON_SOURCES}
     ${CMAKE_CURRENT_SOURCE_DIR}/__init__.py
     PARENT_SCOPE
)

This works, but the building of A_PYTHON_SOURCES using PARENT_SCOPE feels a bit dirty. Another option I know of is to include(...) all the descendant CMakeLists.txt from a/CMakeLists.txt which would avoid PARENT_SCOPE, but then you’re leaking the directory hierarchy into the top level file.

target_sources seems like a good option, but since I need to copy Python source files, and am not building a library or executable, it doesn’t seem possible to use it in my case.

What is the right way to go about this? Is there a better way to manually enumerate a tree of source files than what I’m doing?

Thanks,
Ashish.

2

This feels like an anti-practice. Shouldn’t your sources be controlled?

I don’t know that I would litter my python module tree with CMakeLists.txt files. If you need python files for unit tests, consider either configure_file or modifying the PYTHONPATH environment variable to include this source tree.

3

Yes, all the files shown in the question are under source control, but why does that matter? I’m trying to figure out when any of the Python files have been modified so I can copy them to the binary directory. (I’m not a CMake expert by any means, so I might be missing something that should be obvious)

I agree, I don’t like it either, but the other option I can think of is listing them all in a/CMakeLists.txt, which I don’t like either.

Are you saying use configure_file(... COPYONLY) for each .py file? If so, how do I get the list of files to be copied (which is basically my question)?

I’m not only running unit tests, but the files under ${CMAKE_BINARY_DIR}/py are also being built into an sdist. So I need the directory structure created by copying the files under a to ${CMAKE_BINARY_DIR}/py.

In any case, even if I could use PYTHONPATH or whatever to run unit tests under the source tree, how do I detect when any of the files have changed so I can re-run unit tests? Again, it seems to me I need a list of all Python source files for doing that.

4

It just struck me as odd. Using rdist is stepping outside of CMake, and is extremely unlikely to be portable.

Your choices are either use a glob, or explicitly enumerate. While it feels tedious, explicit enumeration is (I think) generally preferred.

I haven’t made a source distribution, but I would expect you provide these files using something along the lines of install.

The blind leading the visually-impaired…

5

Ah yes, the rsync is an odd choice and unportable, someone else put that in. I should be able to replace it with ${CMAKE_COMMAND} -E copy_directory ... but since portability is not a concern for this project, I haven’t bothered.

I’ve read this too, and that’s what my question is essentially - is there a better way to manually enumerate than what I’m doing? I think I’ve drowned that out with too much verbiage, so I’ll clarify the original question.

That’s an interesting thought, thanks! I’m probably not going to be able to use that though because the sdist is being installed into a virtualenv which is created as part of the build, so that yet another suite of tests can be run. Sorry for moving the goal posts, I didn’t think this was relevant information initially.