Custom language: Designing a thin wrapper for pre-compilers

Code
1

This is related to fypp which has been brought up a few time around here. I’ve been reading about this implementation, and it got me thinking if it’s possible to design a simple language wrapper. I have been considering a few options, but I am having troubles getting either to work:

Common prepare

Either in CMakeDetermineFyppCompiler.cmake or somewhere in an included path, the following are imported:

find_package(Python3 REQUIRED)
cmake_path(GET Python3_EXECUTABLE PARENT_PATH Python3_BINDIR)
find_program(
        CMAKE_Fypp_COMPILER
        NAMES fypp
        HINTS "${Python3_BINDIR}"
        DOC "Fypp preprocessor"
)
mark_as_advanced(CMAKE_Fypp_COMPILER)

set(CMAKE_Fypp_SOURCE_FILE_EXTENSIONS fypp)
set(CMAKE_Fypp_OUTPUT_EXTENSION .f90)
set(CMAKE_Fypp_COMPILER_ENV_VAR "")

Other boilerplates like configure_file(CMakeFyppCompiler.cmake) are left out for brevity

Option A: overwriting CMAKE_Fortran_PREPROCESS_SOURCE as a module

Within the included path:

set(CMAKE_Fortran_SOURCE_FILE_EXTENSIONS "${CMAKE_Fortran_SOURCE_FILE_EXTENSIONS};fypp")
set(CMAKE_Fortran_PREPROCESS_SOURCE "<CMAKE_Fypp_COMPILER> <SOURCE> <PREPROCESSED_SOURCE>")

This does not seem to work because the file suffix in <PREPROCESSED_SOURCE> is not changed and the compiler is unable to detect that it is a Fortran source file.

Is there a way to alter <PREPROCESSED_SOURCE> (as a rule, not individually) so that it is properly consumed down the line, e.g. by CMAKE_Fortran_COMPILE_OBJECT.

Note: it is surprising that that one is picked up, but CMAKE_Fortran_CREATE_PREPROCESS_SOURCE which is defined in the source files is not.

Option B: defining Fypp language

The CMakeFyppInformation.cmake file looks generally like this

set(CMAKE_Fypp_CREATE_PREPROCESS_SOURCE "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <PREPROCESSED_SOURCE>")
if (NOT CMAKE_Fypp_CREATE_SHARED_LIBRARY)
    set(CMAKE_Fypp_CREATE_SHARED_LIBRARY "${CMAKE_Fortran_CREATE_SHARED_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_MODULE)
    set(CMAKE_Fypp_CREATE_SHARED_MODULE "${CMAKE_Fortran_CREATE_SHARED_MODULE}")
endif ()
if (NOT CMAKE_Fypp_CREATE_STATIC_LIBRARY)
    set(CMAKE_Fypp_CREATE_STATIC_LIBRARY "${CMAKE_Fortran_CREATE_STATIC_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_COMPILE_OBJECT)
    set(CMAKE_Fypp_COMPILE_OBJECT
            "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <OBJECT>"
            "${CMAKE_Fortran_COMPILE_OBJECT}")
endif ()
if (NOT CMAKE_Fypp_LINK_EXECUTABLE)
    set(CMAKE_Fypp_LINK_EXECUTABLE "${CMAKE_Fortran_LINK_EXECUTABLE}")
endif ()
set(CMAKE_Fypp_INFORMATION_LOADED 1)

I have tried a few variations around it but with no success. Some of the issues I encountered:

  • When expanding CMAKE_Fortran_COMPILE_OBJECT, the <OBJECT> etc. variables are expanded with respect to the definition of Fypp language. Is there a way to generator-expression the variables to a different language?
  • Is there a way to inject the “object” files of Fypp to be source files for the Fotran language? Ideally that would be cleanest since it would only need to define a CMAKE_Fypp_COMPILE_OBJECT command structure.
  • What is missing that CMAKE_Fypp_CREATE_PREPROCESS_SOURCE is not being picked up? It seems that it is hard-coded?

A similar usecase for this is in swig in order to simplify the interface.

2

I managed to get a bit further by using this debug language CMakeDebugInformation.cmake:

set(print_metadata
        "\
        CMAKE_Debug_COMPILER = <CMAKE_Debug_COMPILER>\t\
        SOURCE = <SOURCE>\t\
        OBJECT = <OBJECT>\t\
        OBJECT_DIR = <OBJECT_DIR>\t\
        OBJECT_FILE_DIR = <OBJECT_FILE_DIR>\t\
        OBJECTS = <OBJECTS>\t\
        OBJECTS_QUOTED = <OBJECTS_QUOTED>\t\
        DEFINES = <DEFINES>\t\
        INCLUDES = <INCLUDES>\t\
        MANIFESTS = <MANIFESTS>\t\
        FLAGS = <FLAGS>\t\
        LINK_FLAGS = <LINK_FLAGS>\t\
        DEP_FILE = <DEP_FILE>\t\
        DEP_TARGET = <DEP_TARGET>\t\
        DYNDEP_FILE = <DYNDEP_FILE>\t\
        TARGET_BASE = <TARGET_BASE>\t\
        TARGET = <TARGET>\t\
        TARGET_QUOTED = <TARGET_QUOTED>\t\
        TARGET_UNQUOTED = <TARGET_UNQUOTED>\t\
        LANGUAGE_COMPILE_FLAGS = <LANGUAGE_COMPILE_FLAGS>\t\
        PREPROCESSED_SOURCE = <PREPROCESSED_SOURCE>\t\
        ASSEMBLY_SOURCE = <ASSEMBLY_SOURCE>\t\
        "
        )

set(CMAKE_Debug_CREATE_PREPROCESS_SOURCE
        "echo \"Create_PREPROCESS_SOURCE:: ${print_metadata}\"")
set(CMAKE_Debug_PREPROCESS_SOURCE
        "echo \"PREPROCESS_SOURCE:: ${print_metadata}\"")
set(CMAKE_Debug_COMPILE_OBJECT
        "echo \"COMPILE_OBJECT:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_SHARED_LIBRARY
        "echo \"CREATE_SHARED_LIBRARY:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_SHARED_MODULE
        "echo \"CREATE_SHARED_MODULE:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_STATIC_LIBRARY
        "echo \"CREATE_STATIC_LIBRARY:: ${print_metadata}\"")
set(CMAKE_Debug_LINK_EXECUTABLE
        "echo \"LINK_EXECUTABLE:: ${print_metadata}\"")

Evidently, not all variables are expanded, depending on what calls it, e.g. LINK_EXECUTABLE has TARGET but no SOURCE, while COMPILE_OBJECT has the reverse.

A bit bothersome is that PREPROCESSED_SOURCE is not accessible or any bare form of the files, but it is still manageable somehow. Here’s a working approach to Fypp “language” if someone needs some inspiration.

if (NOT CMAKE_Fypp_COMPILE_OBJECT)
    string(REPLACE "<SOURCE>" "<OBJECT>.pp.f90" _Fypp_Fortran_COMPILE_OBJECT "${CMAKE_Fortran_COMPILE_OBJECT}")
    set(CMAKE_Fypp_COMPILE_OBJECT
            # Precompile with fypp
            "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <OBJECT>.pp.f90"
            # Compile object with fortran compiler using the pre-compiled source
            "${_Fypp_Fortran_COMPILE_OBJECT}"
            )
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_LIBRARY)
    set(CMAKE_Fypp_CREATE_SHARED_LIBRARY "${CMAKE_Fortran_CREATE_SHARED_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_MODULE)
    set(CMAKE_Fypp_CREATE_SHARED_MODULE "${CMAKE_Fortran_CREATE_SHARED_MODULE}")
endif ()
if (NOT CMAKE_Fypp_CREATE_STATIC_LIBRARY)
    if (DEFINED CMAKE_Fortran_CREATE_STATIC_LIBRARY)
        set(CMAKE_Fypp_CREATE_STATIC_LIBRARY "${CMAKE_Fortran_CREATE_STATIC_LIBRARY}")
    else ()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_CREATE)
            set(CMAKE_Fypp_ARCHIVE_CREATE "${CMAKE_Fortran_ARCHIVE_CREATE}")
        endif()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_APPEND)
            set(CMAKE_Fypp_ARCHIVE_APPEND "${CMAKE_Fortran_ARCHIVE_APPEND}")
        endif()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_FINISH)
            set(CMAKE_Fypp_ARCHIVE_FINISH "${CMAKE_Fortran_ARCHIVE_FINISH}")
        endif()
    endif ()
endif ()
if (NOT CMAKE_Fypp_LINK_EXECUTABLE)
    set(CMAKE_Fypp_LINK_EXECUTABLE "${CMAKE_Fortran_LINK_EXECUTABLE}")
endif ()

set(CMAKE_Fypp_INFORMATION_LOADED 1)

This still needs work because it does not properly append the fortran modules to cmake files.

Note: <SOURCE>- fails to expand, but <SOURCE>. can expand, so we cannot exactly mimick cmake’s syntax of -pp.f90 precompiled files

Also what’s the purpose of DYNDEP_FILE and how can it be used?

3

It is used to compute the order of compilation due to Fortran modules declared and used in various sources. It is gated on the language being Fortran and cannot be accessed any other way (as is all other module support code). CMake does not have the abstractions for module support in arbitrary languages.

4

I am coming back to this topic, now at the angle of cython (experimentation here) which avoids the module logic. I am considering 2 approaches

Approach 1: set(CMAKE_Cython_OUTPUT_EXTENSION ".c")

The idea of this is to create the most minimal CMake<Lang>Information.cmake file as:

set(CMAKE_Cython_COMPILE_OBJECT
        "<CMAKE_Cython_COMPILER> <INCLUDES> <FLAGS> -o <OBJECT> <SOURCE>")
set(CMAKE_Cython_INFORMATION_LOADED 1)

and let the C compiler take over from there.

Unfortunately this does not seem to work because the generated C file does not pick up that it has source files generated. Maybe with some tweaks to CMake where you would define something like CMAKE_<Lang>_FORWARD_COMPILER and then whenever a source is generated on <LANG> , append <OBJECT> as an additional source with the language defined in CMAKE_<Lang>_FORWARD_COMPILER.

Approach 2: chain CMAKE_Cython_COMPILE_OBJECT with CMAKE_C_COMPILE_OBJECT

As an intermediary until the other approach is explored, this seems to work, except for a few caveats to further explore. This approach looks something like

## Inherit various definitions from C compiler, e.g.
set(CMAKE_Cython_COMPILE_OPTIONS_PIC ${CMAKE_C_COMPILE_OPTIONS_PIC})

## Main definition
string(REPLACE "<SOURCE>" "<OBJECT>.c" CMAKE_C_COMPILE_OBJECT_replaced "${CMAKE_C_COMPILE_OBJECT}")
set(CMAKE_Cython_COMPILE_OBJECT
        "<CMAKE_Cython_COMPILER> -o <OBJECT>.c <SOURCE>"
        "${CMAKE_C_COMPILE_OBJECT_replaced}"

The few issues I am encountering:

  • For some reason the library file suffix became .so.so, probably an interference with Python_add_library
  • How to expose and handle dependencies?