Potential regression?: CMake + SWIG not detecting C++ headers as dependencies

Hello everyone.

I am looking for help in either configuring properly my project or determining if this is a bug that should be reported on gitlab. I have prepared a minimalistic project that reproduces the issue (but as a new user I cannot attach it… uhhh… I pasted all contents at the end).

The issue is that after performing changes in header files that SWIG template inlcudes (like %include mylib.hpp), the autogenerated SWIG wrapper is not re-generated and resulting module is not recompiled.

If you take my code, create a build directory, navigate to it and execute make check, a failing test will get executed. It fails with SWIG error which says that value provided to function is too large and it overflows int8_t.

Hello world
error: in method 'add1', argument 1 of type 'int8_t' (SWIG_OverflowError)

To make it pass, you should edit mylib.cpp/hpp and change int8_t in function add1 to uint8_t, both in argument and return value.

After you make this change, you should expect that autogenerated SWIG module would be re-generated and resulting code re-compiled to reflect this change. Unfortunatelly, this is not what happens - running the test case again with make check recompiles target libmylib but not mylib_octave, so it prints the same kind of error about overflowing int8_t (where at this point it should be uint8_t and there should be no overflow).

If you now execute make clean && make check, the test passes.

This issue does not happen if I provide relative paths to included files in mylib_octave.swg, like if I write %include ../src/mylib.hpp instead of simple %include mylib.hpp. Following this trail, I’ve noticed that using relative paths to included file properly adds this file to build/test/CMakeFiles/mylib_octave_swig_compilation.dir/depend.make as a dependency of .stamp file. This suggests that SWIG module/wrapper generation is not properly using include directories that I have configured, or I did this wrong.

I am using CMake version 3.16.4, but so far didn’t find any substantial changes in recent versions in UseSWIG module that would help with my issue.

The directory tree is as follows:

cmake_bug_mvp
│   CMakeLists.txt
│
├───cmake
│       FindOctave.cmake
│
├───include
│       external_interface_here
│
├───src
│       mylib.cpp
│       mylib.hpp
│
└───test
        CMakeLists.txt
        mylib_octave.swg
        run_octave.sh
        test_mylib.m
Contents of files

./CMakeLists.txt

cmake_minimum_required(VERSION 3.16.3)
project(cmake_bug_mvp VERSION 0.1.0)

list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")

set(CMAKE_C_STANDARD 11)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_subdirectory(test EXCLUDE_FROM_ALL)

add_library(mylib
${CMAKE_CURRENT_SOURCE_DIR}/src/mylib.cpp
)

target_include_directories(mylib
PUBLIC include
PRIVATE src
)

include(CheckPIESupported)
check_pie_supported()
set_property(TARGET mylib PROPERTY POSITION_INDEPENDENT_CODE TRUE)

src/mylib.hpp

#pragma once

#include <cstdint>

void hello(char * who);
int8_t add1(int8_t x);

src/mylib.cpp

#include "mylib.hpp"

#include <iostream>

void hello(char * who) {
    std::cout << "Hello " << who << std::endl;
}

int8_t add1(int8_t x){
    return x+1;
}

test/CMakeLists.txt

find_package(Octave)
find_package(SWIG 4.0)
cmake_policy(SET CMP0078 NEW)
include(UseSWIG)
set(UseSWIG_MODULE_VERSION 2)
set(UseSWIG_TARGET_NAME_PREFERENCE STANDARD)
set(SWIG_SOURCE_FILE_EXTENSIONS ".i" ".swg")

set_property(SOURCE mylib_octave.swg PROPERTY CPLUSPLUS ON)
swig_add_library(mylib_octave
        LANGUAGE Octave
        SOURCES mylib_octave.swg
)
set_property(TARGET mylib_octave PROPERTY SWIG_USE_TARGET_INCLUDE_DIRECTORIES TRUE)
set_property(TARGET mylib_octave PROPERTY SWIG_GENERATED_INCLUDE_DIRECTORIES ${OCTAVE_INCLUDE_DIR}/../)
target_link_libraries(mylib_octave mylib)
target_include_directories(mylib_octave
        PRIVATE ../src
)

enable_testing()
add_executable(run_octave IMPORTED)
set_property(TARGET run_octave PROPERTY IMPORTED_LOCATION ${CMAKE_CURRENT_SOURCE_DIR}/run_octave.sh)
add_dependencies(run_octave mylib_octave)

function(add_octave_test name)
        add_test(NAME ${name}
                COMMAND run_octave ${CMAKE_CURRENT_SOURCE_DIR}/${name}.m
        )
endfunction()

add_octave_test(test_mylib)

add_custom_target(
        check COMMAND ${CMAKE_CTEST_COMMAND} --output-on-failure
        DEPENDS run_octave
)

test/mylib_octave.swg

%module mylib_octave
%{
#include "mylib.hpp"
#define SWIG_OCTAVE_NO_SEGFAULT_HACK
%}
%include "stdint.i"
%include "typemaps.i"
%include "mylib.hpp"

test/run_octave.sh

#!/usr/bin/env bash

octave-cli -qfH "$1"
[ $? -eq 0 ]  || exit 1

test/test_mylib.m

mylib_octave
hello("world")
add1(254)
exit(0)

I am afraid the dependency management didn’t work because the swig file is compiled using the add_custom_command() command and include directories are not managed for this kind of target.

If you specify a relative path to your include path, it works because the CMake scanner is activated on the swig source file due to the IMPLICIT_DEPENDS clause specified on the add_custom_command().

You can solve your problem by explicitly specify the dependency by using source property DEPENDS:

set_property(SOURCE mylib_octave.swg PROPERTY DEPENDS ../src/mylib.hpp)

Hi, thank you for your answer.

While the workaround is, well, working, I will have trouble maintaining it for a larger project, caring for each and every header file used by SWIG to be listed there.One missed header and I am back to the situation which forced me to write the previous post.

Following your explanation I struggle to understand what was the intention of all the code in UseSWIG.cmake and CMake itself that tries to parse the SWIG templates to track the dependencies (lines beginning with % or #) and effecitvely fails.

See, in the way you put it, there are some undocumented limitations on how CMake can track dependencies on files included from SWIG module, namely: the file must be included with either absolute or relative path to the file.

This is effectively disabling any ability to reuse already written C/C++ headers in SWIG modules as they might include other files without relative path specified - they assume that include_directories takes care of providing correct path [for the complier] to look for header files.

So the question here is, why this line in UseSWIG.cmake:

IMPLICIT_DEPENDS CXX "${swig_source_file_fullname}"

doesn’t work as advertised, i.e. it produces non-existent paths to files in depend.make and does not consider any kind of configuration option similar to include_directories:

IMPLICIT_DEPENDS
Request scanning of implicit dependencies of an input file. The language given specifies the programming language whose corresponding dependency scanner should be used. Currently only C and CXX language scanners are supported. The language has to be specified for every file in the IMPLICIT_DEPENDS list. Dependencies discovered from the scanning are added to those of the custom command at build time. Note that the IMPLICIT_DEPENDS option is currently supported only for Makefile generators and will be ignored by other generators.

For me this only brings inconsistency and confusion, IMHO the dependency scanning should just work or not be here at all :confused:. Or am I missing something?

On the other hand, if I work-around this dependency scanning by adding all header files in src and include directories by globbing, IMHO this just might work as well as regenerating the module on every occasion and be golden, why care for dependencies…

As I already said, IMPLICIT_DEPENDS activate the parsing of dependencies on the file specified but, because custom target does not have the concept of include directories, there is a limitation on the capabilities of discovery of dependencies.
This is a limitation of the current implementation of IMPLICIT_DEPENDS.

I am splitting my posts due to:

Sorry, new users can only put 2 links in a post.

============================================

Ok then, for future reference and others interested (if there are any):
My issue seems to be an exact copy of two stackoverflow questions from 4 and 5 years ago respectively.

They reference an older version of UseSWIG.cmake module, but the issue is exactly the same.

The feature that I am after was initially implemented here:
https://public.kitware.com/Bug/view.php?id=4147

Next it was reverted due to this:
https://public.kitware.com/Bug/view.php?id=12307

And the continuation of this topic is in the issue 4147 on gitlab, reopened 3 years ago that references two closed merge requests which were supposed to change something on this matter.

https://gitlab.kitware.com/cmake/cmake/-/issues/4147

Brad commented:

!354 (merged) contains a solution that works for Makefile generators but not other generators.

Well it looks to me that it acutally does not even work consistently for Makefile generators (see my previous posts and Marc’s answer).

I would just like to know if trying to re-implement the initial patch from 2006 (with ignore-missing added) is time-worthy for CMake maintainers, or the current state is how it should work and I just should not waste your time anymore (hey, at least I have a workaround!).

Another question is if, at least temporarily, all of this information should be added somewhere to the documentation, so other people that find themselves in my shoes don’t have to do this research all over again. :wink:

I think the right approach is effectively to rely on flags -M* to compute the dependencies.

FYI, I am currently working on enhancements to dependencies management (Makefiles compiler dependencies) for makefiles generators by relying on compiler itself (i.e. options -M family on gcc for example) rather than the CMake scanner.

I think it will be possible to extend add_custom_command to support this new approach… And also to extend the support to Ninja which rely natively on gcc style for dependencies.

Feel free to create an issue to request documentation clarification. And you can also create an MR to enhance the documentation. :slight_smile:

1 Like