# Custom language: Designing a thin wrapper for pre-compilers

**URL:** https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456
**Category:** Code
**Created:** [July 4, 2023, 11:46am UTC](https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456 "2023-07-04T11:46:05Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Lecris](https://discourse.cmake.org/user_avatar/discourse.cmake.org/lecris/32/3193_2.png) [@Lecris](https://discourse.cmake.org/u/Lecris)
#### Post date: [July 4, 2023, 11:46am UTC](https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456/1 "2023-07-04T11:46:05Z")

</div>

This is related to `fypp` which has been brought up a few time around here. I’ve been reading about [this implementation](https://stackoverflow.com/questions/63658231/add-an-extra-compile-step-to-a-custom-compiler-language-in-cmake), and it got me thinking if it’s possible to design a simple language wrapper. I have been considering a few options, but I am having troubles getting either to work:

> **Common prepare**
>
> Either in `CMakeDetermineFyppCompiler.cmake` or somewhere in an included path, the following are imported:
> 
> ```cmake
> find_package(Python3 REQUIRED)
> cmake_path(GET Python3_EXECUTABLE PARENT_PATH Python3_BINDIR)
> find_program(
> CMAKE_Fypp_COMPILER
> NAMES fypp
> HINTS "${Python3_BINDIR}"
> DOC "Fypp preprocessor"
> )
> mark_as_advanced(CMAKE_Fypp_COMPILER)
> 
> set(CMAKE_Fypp_SOURCE_FILE_EXTENSIONS fypp)
> set(CMAKE_Fypp_OUTPUT_EXTENSION .f90)
> set(CMAKE_Fypp_COMPILER_ENV_VAR "")
> 
> ```
> 
> Other boilerplates like `configure_file(CMakeFyppCompiler.cmake)` are left out for brevity

### Option A: overwriting `CMAKE_Fortran_PREPROCESS_SOURCE` as a module

Within the included path:

```cmake
set(CMAKE_Fortran_SOURCE_FILE_EXTENSIONS "${CMAKE_Fortran_SOURCE_FILE_EXTENSIONS};fypp")
set(CMAKE_Fortran_PREPROCESS_SOURCE "<CMAKE_Fypp_COMPILER> <SOURCE> <PREPROCESSED_SOURCE>")

```

This does not seem to work because the file suffix in `<PREPROCESSED_SOURCE>` is not changed and the compiler is unable to detect that it is a Fortran source file.

Is there a way to alter `<PREPROCESSED_SOURCE>` (as a rule, not individually) so that it is properly consumed down the line, e.g. by `CMAKE_Fortran_COMPILE_OBJECT`.

Note: it is surprising that that one is picked up, but `CMAKE_Fortran_CREATE_PREPROCESS_SOURCE` which is defined in the source files is not.

### Option B: defining `Fypp` language

The `CMakeFyppInformation.cmake` file looks generally like this

```cmake
set(CMAKE_Fypp_CREATE_PREPROCESS_SOURCE "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <PREPROCESSED_SOURCE>")
if (NOT CMAKE_Fypp_CREATE_SHARED_LIBRARY)
    set(CMAKE_Fypp_CREATE_SHARED_LIBRARY "${CMAKE_Fortran_CREATE_SHARED_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_MODULE)
    set(CMAKE_Fypp_CREATE_SHARED_MODULE "${CMAKE_Fortran_CREATE_SHARED_MODULE}")
endif ()
if (NOT CMAKE_Fypp_CREATE_STATIC_LIBRARY)
    set(CMAKE_Fypp_CREATE_STATIC_LIBRARY "${CMAKE_Fortran_CREATE_STATIC_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_COMPILE_OBJECT)
    set(CMAKE_Fypp_COMPILE_OBJECT
            "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <OBJECT>"
            "${CMAKE_Fortran_COMPILE_OBJECT}")
endif ()
if (NOT CMAKE_Fypp_LINK_EXECUTABLE)
    set(CMAKE_Fypp_LINK_EXECUTABLE "${CMAKE_Fortran_LINK_EXECUTABLE}")
endif ()
set(CMAKE_Fypp_INFORMATION_LOADED 1)

```

I have tried a few variations around it but with no success. Some of the issues I encountered:

- When expanding `CMAKE_Fortran_COMPILE_OBJECT`, the `<OBJECT>` etc. variables are expanded with respect to the definition of `Fypp` language. Is there a way to generator-expression the variables to a different language?
- Is there a way to inject the “object” files of `Fypp` to be source files for the Fotran language? Ideally that would be cleanest since it would only need to define a `CMAKE_Fypp_COMPILE_OBJECT` command structure.
- What is missing that `CMAKE_Fypp_CREATE_PREPROCESS_SOURCE` is not being picked up? It seems that it is [hard-coded](https://github.com/Kitware/CMake/blob/7bd24c1da5d093199dedab3fcfad6ec93d4c7aa5/Source/cmMakefileTargetGenerator.cxx#L963)?

* * *

A similar usecase for this is in `swig` in order to simplify the interface.

---

<div class="post-metadata">

### Author: ![Lecris](https://discourse.cmake.org/user_avatar/discourse.cmake.org/lecris/32/3193_2.png) [@Lecris](https://discourse.cmake.org/u/Lecris)
#### Post date: [July 5, 2023, 9:53am UTC](https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456/2 "2023-07-05T09:53:27Z")

</div>

I managed to get a bit further by using this debug language `CMakeDebugInformation.cmake`:

```cmake
set(print_metadata
        "\
        CMAKE_Debug_COMPILER = <CMAKE_Debug_COMPILER>\t\
        SOURCE = <SOURCE>\t\
        OBJECT = <OBJECT>\t\
        OBJECT_DIR = <OBJECT_DIR>\t\
        OBJECT_FILE_DIR = <OBJECT_FILE_DIR>\t\
        OBJECTS = <OBJECTS>\t\
        OBJECTS_QUOTED = <OBJECTS_QUOTED>\t\
        DEFINES = <DEFINES>\t\
        INCLUDES = <INCLUDES>\t\
        MANIFESTS = <MANIFESTS>\t\
        FLAGS = <FLAGS>\t\
        LINK_FLAGS = <LINK_FLAGS>\t\
        DEP_FILE = <DEP_FILE>\t\
        DEP_TARGET = <DEP_TARGET>\t\
        DYNDEP_FILE = <DYNDEP_FILE>\t\
        TARGET_BASE = <TARGET_BASE>\t\
        TARGET = <TARGET>\t\
        TARGET_QUOTED = <TARGET_QUOTED>\t\
        TARGET_UNQUOTED = <TARGET_UNQUOTED>\t\
        LANGUAGE_COMPILE_FLAGS = <LANGUAGE_COMPILE_FLAGS>\t\
        PREPROCESSED_SOURCE = <PREPROCESSED_SOURCE>\t\
        ASSEMBLY_SOURCE = <ASSEMBLY_SOURCE>\t\
        "
        )

set(CMAKE_Debug_CREATE_PREPROCESS_SOURCE
        "echo \"Create_PREPROCESS_SOURCE:: ${print_metadata}\"")
set(CMAKE_Debug_PREPROCESS_SOURCE
        "echo \"PREPROCESS_SOURCE:: ${print_metadata}\"")
set(CMAKE_Debug_COMPILE_OBJECT
        "echo \"COMPILE_OBJECT:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_SHARED_LIBRARY
        "echo \"CREATE_SHARED_LIBRARY:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_SHARED_MODULE
        "echo \"CREATE_SHARED_MODULE:: ${print_metadata}\"")
set(CMAKE_Debug_CREATE_STATIC_LIBRARY
        "echo \"CREATE_STATIC_LIBRARY:: ${print_metadata}\"")
set(CMAKE_Debug_LINK_EXECUTABLE
        "echo \"LINK_EXECUTABLE:: ${print_metadata}\"")

```

Evidently, not all variables are expanded, depending on what calls it, e.g. `LINK_EXECUTABLE` has `TARGET` but no `SOURCE`, while `COMPILE_OBJECT` has the reverse.

A bit bothersome is that `PREPROCESSED_SOURCE` is not accessible or any bare form of the files, but it is still manageable somehow. Here’s a working approach to `Fypp` “language” if someone needs some inspiration.

```cmake
if (NOT CMAKE_Fypp_COMPILE_OBJECT)
    string(REPLACE "<SOURCE>" "<OBJECT>.pp.f90" _Fypp_Fortran_COMPILE_OBJECT "${CMAKE_Fortran_COMPILE_OBJECT}")
    set(CMAKE_Fypp_COMPILE_OBJECT
            # Precompile with fypp
            "<CMAKE_Fypp_COMPILER> <DEFINES> <INCLUDES> <SOURCE> <OBJECT>.pp.f90"
            # Compile object with fortran compiler using the pre-compiled source
            "${_Fypp_Fortran_COMPILE_OBJECT}"
            )
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_LIBRARY)
    set(CMAKE_Fypp_CREATE_SHARED_LIBRARY "${CMAKE_Fortran_CREATE_SHARED_LIBRARY}")
endif ()
if (NOT CMAKE_Fypp_CREATE_SHARED_MODULE)
    set(CMAKE_Fypp_CREATE_SHARED_MODULE "${CMAKE_Fortran_CREATE_SHARED_MODULE}")
endif ()
if (NOT CMAKE_Fypp_CREATE_STATIC_LIBRARY)
    if (DEFINED CMAKE_Fortran_CREATE_STATIC_LIBRARY)
        set(CMAKE_Fypp_CREATE_STATIC_LIBRARY "${CMAKE_Fortran_CREATE_STATIC_LIBRARY}")
    else ()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_CREATE)
            set(CMAKE_Fypp_ARCHIVE_CREATE "${CMAKE_Fortran_ARCHIVE_CREATE}")
        endif()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_APPEND)
            set(CMAKE_Fypp_ARCHIVE_APPEND "${CMAKE_Fortran_ARCHIVE_APPEND}")
        endif()
        if(NOT DEFINED CMAKE_Fypp_ARCHIVE_FINISH)
            set(CMAKE_Fypp_ARCHIVE_FINISH "${CMAKE_Fortran_ARCHIVE_FINISH}")
        endif()
    endif ()
endif ()
if (NOT CMAKE_Fypp_LINK_EXECUTABLE)
    set(CMAKE_Fypp_LINK_EXECUTABLE "${CMAKE_Fortran_LINK_EXECUTABLE}")
endif ()

set(CMAKE_Fypp_INFORMATION_LOADED 1)

```

This still needs work because it does not properly append the fortran modules to cmake files.

Note: `<SOURCE>-` fails to expand, but `<SOURCE>.` can expand, so we cannot exactly mimick cmake’s syntax of `-pp.f90` precompiled files

* * *

Also what’s the purpose of `DYNDEP_FILE` and how can it be used?

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [July 9, 2023, 11:10am UTC](https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456/3 "2023-07-09T11:10:39Z")

</div>

> [@Lecris](#):
>
> Also what’s the purpose of `DYNDEP_FILE` and how can it be used?

It is used to compute the order of compilation due to Fortran modules declared and used in various sources. It is gated on the language being `Fortran` and cannot be accessed any other way (as is all other module support code). CMake does not have the abstractions for module support in arbitrary languages.

---

<div class="post-metadata">

### Author: ![Lecris](https://discourse.cmake.org/user_avatar/discourse.cmake.org/lecris/32/3193_2.png) [@Lecris](https://discourse.cmake.org/u/Lecris)
#### Post date: [August 26, 2024, 6:58am UTC](https://discourse.cmake.org/t/custom-language-designing-a-thin-wrapper-for-pre-compilers/8456/4 "2024-08-26T06:58:12Z")

</div>

I am coming back to this topic, now at the angle of `cython` (experimentation [here](https://github.com/scikit-build/cython-cmake/pull/48)) which avoids the `module` logic. I am considering 2 approaches

### Approach 1: `set(CMAKE_Cython_OUTPUT_EXTENSION ".c")`

The idea of this is to create the most minimal `CMake<Lang>Information.cmake` file as:

```cmake
set(CMAKE_Cython_COMPILE_OBJECT
        "<CMAKE_Cython_COMPILER> <INCLUDES> <FLAGS> -o <OBJECT> <SOURCE>")
set(CMAKE_Cython_INFORMATION_LOADED 1)

```

and let the `C` compiler take over from there.

Unfortunately this does not seem to work because the generated `C` file does not pick up that it has source files generated. Maybe with some tweaks to CMake where you would define something like `CMAKE_<Lang>_FORWARD_COMPILER` and then whenever a source is generated on `<LANG>` , append `<OBJECT>` as an additional source with the language defined in `CMAKE_<Lang>_FORWARD_COMPILER`.

### Approach 2: chain `CMAKE_Cython_COMPILE_OBJECT` with `CMAKE_C_COMPILE_OBJECT`

As an intermediary until the other approach is explored, this seems to work, except for a few caveats to further explore. This approach looks something like

```cmake
## Inherit various definitions from C compiler, e.g.
set(CMAKE_Cython_COMPILE_OPTIONS_PIC ${CMAKE_C_COMPILE_OPTIONS_PIC})

## Main definition
string(REPLACE "<SOURCE>" "<OBJECT>.c" CMAKE_C_COMPILE_OBJECT_replaced "${CMAKE_C_COMPILE_OBJECT}")
set(CMAKE_Cython_COMPILE_OBJECT
        "<CMAKE_Cython_COMPILER> -o <OBJECT>.c <SOURCE>"
        "${CMAKE_C_COMPILE_OBJECT_replaced}"

```

The few issues I am encountering:

- For some reason the library file suffix became `.so.so`, probably an interference with `Python_add_library`
- How to expose and handle dependencies?