Using C++20 modules in a multiple directory project

nitroglyceryne · October 22, 2023, 3:55pm

Hey everyone !

I wanted to try C++ modules on a project I’m working on. This project contains multiple directories and multiple CMakeLists.txt files. The structure is as follows:

Root/
├─ Kernel/
│  ├─ CMakeLists.txt
│  ├─ BootInfo/
│  │  ├─ CMakeLists.txt
│  │  ├─ BootInfo.cppm
│  ├─ Graphics/
│  │  ├─ CMakeLists.txt
│  │  ├─ FrameBuffer.cppm
├─ PdSTL/
│  ├─ CMakeLists.txt
│  ├─ Variant/
│  │  ├─ CMakeLists.txt
│  │  ├─ VariantStorage.cppm
│  │  ├─ VariantChoice.cppm
│  ├─ Variant.cppm
├─ CMakeLists.txt
├─ src/
│  ├─ Kernel.cpp

The content of the PdSTL directory files is as follows:

Root/PdSTL/Variant.cppm:

export module PdSTL.Variant;

import :VariantStorage;
import :VariantChoice;

export template <typename... Types>
class Variant : private VariantStorage<Types...>,
                private  VariantChoice<Types, Types...>... {

//variant sturff
};

Root/PdSTL/Variant/VariantStorage.cppm:

export module PdSTL.Variant:VariantStorage;

template <typename... Types>
class VariantStorage {
   // Internal implementation of variant storage
   // This partition is not exported here or in primary interface unit
   //  as it is an implementation detail...
};

Root/PdSTL/Variant/VariantChoice.cppm:

export module PdSTL.Variant:VariantChoice;

template <typename T, typename... Types>
class VariantChoice {
  // Internal implementation of variant choice
  // This partition is not exported here or in primary interface unit 
  // as it is an implementation detail...
};

CMakeLists.txt files for PdSTL directory is:

Root/PdSTL/Variant/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28)

target_sources(PdSTL PUBLIC FILE_SET pdstl___variant TYPE CXX_MODULES FILES
               ${CMAKE_CURRENT_SOURCE_DIR}/variant_storage.cppm
               ${CMAKE_CURRENT_SOURCE_DIR}/variant_choice.cppm)

Root/PdSTL/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28)
project(PdSTL CXX)
set(CMAKE_CXX_STANDARD 23)

# Default to C++ extensions being off. Clang's modules support have trouble
# with extensions right now and it is not required for any other compiler
set(CMAKE_CXX_EXTENSIONS OFF)

add_library(PdSTL STATIC "")
add_subdirectory(Variant)

target_sources(PdSTL PUBLIC FILE_SET pd_stl_root TYPE CXX_MODULES FILES
               Variant.cppm)

set_target_properties(PdSTL PROPERTIES COMPILE_FLAGS "-g -Wall -Werror -Wextra -Wpedantic -ffreestanding -fno-stack-protector -fno-rtti -fno-exceptions")
target_include_directories(PdSTL PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

The content of the Kernel directory is as follows:

Root/Kernel/BootInfo/BootInfo.cppm:

// Root/Kernel/BootInfo/BootInfo.cppm
export module Kernel.BootInfo;

struct BootInfo {
  // members of BootInfo
};

Root/Kernel/Graphics/FrameBuffer.cppm:

// Root/Kernel/Graphics/FrameBuffer.cppm

export module Kernel.Graphics.FrameBuffer;

struct FrameBuffer {
  // members of FrameBuffer
};

CMakeLists.txt files for Kernel directory are as follows:

Root/Kernel/BootInfo/CMakeLists.txt:

// Root/Kernel/BootInfo/CMakeLists.txt
cmake_minimum_required(VERSION 3.28)

target_sources(Kernel PUBLIC FILE_SET kernel_bootinfo TYPE CXX_MODULES FILES
               ${CMAKE_CURRENT_SOURCE_DIR}/BootInfo.cppm)

target_include_directories(Kernel PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

Root/Kernel/Graphics/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28)

target_sources(Kernel PUBLIC FILE_SET kernel_graphics TYPE CXX_MODULES FILES
               ${CMAKE_CURRENT_SOURCE_DIR}/FrameBuffer.cppm)

target_include_directories(Kernel PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

Root/Kernel/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28)
project(Kernel CXX)
set(CMAKE_CXX_STANDARD 23)


# Default to C++ extensions being off. Clang's modules support have trouble
# with extensions right now and it is not required for any other compiler
set(CMAKE_CXX_EXTENSIONS OFF)

add_library(Kernel "")
add_subdirectory(BootInfo)
add_subdirectory(Graphics)

target_link_libraries(Kernel PdSTL)
set_target_properties(Kernel PROPERTIES COMPILE_FLAGS "-g -Wall -Werror -Wextra -Wpedantic -ffreestanding -fno-stack-protector -fno-rtti -fno-exceptions")

target_include_directories(Kernel PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

Root/Kernel.cpp:

import Kernel.Graphics.FrameBuffer;

int main() {
  // some stuff....
  return 0;
}

Finally, CMakeLists.txt in the Root folder looks as follows:
Root/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28)
project(Root)

set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_EXTENSIONS OFF)

add_executable(Root src/Kernel.cpp)
add_subdirectory(PdSTL)
add_subdirectory(Kernel)
target_link_libraries(Root PdSTL Kernel)

set_target_properties(Root PROPERTIES COMPILE_FLAGS "-g -mcmodel=large -Wall -Werror -Wextra -Wpedantic -ffreestanding -fno-stack-protector -fno-rtti -fno-exceptions -fno-pie -O0")
set_target_properties(Root PROPERTIES LINK_FLAGS "-no-pie -T ${LDS} -static -Bsymbolic -nostdlib")
set_target_properties(Root PROPERTIES OUTPUT_NAME kernel.elf)

With this setup, IDE (CLion) ran the following commands to build the executable:

//When loading CMake project

/usr/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -G Ninja -S /home/user/CLionProjects/Root -B /home/user/CLionProjects/Root/cmake-build-debug

Project loading runs successfully with the following output:

-- The C compiler identification is Clang 16.0.6
-- The CXX compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /usr/bin/clang

-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/user/CLionProjects/Root/

[Finished]

Pressing the build button runs the following command which fails

/usr/bin/cmake --build /home/user/CLionProjects/Root/cmake-build-debug --target Root -j 6

[1/199] Building CXX object CMakeFiles/Root.dir/src/Kernel.cpp.o
FAILED: CMakeFiles/Root.dir/src/Kernel.cpp.o 
/usr/bin/clang++  -I/home/user/CLionProjects/Root/PdSTL -I/home/user/CLionProjects/Root/Kernel/ASM -I/home/user/CLionProjects/Root/Kernel/Base -I/home/user/CLionProjects/Root/Kernel/BootInfo -I/home/user/CLionProjects/Root/Kernel/Debug -I/home/user/CLionProjects/Root/Kernel/EFI_Memory -I/home/user/CLionProjects/Root/Kernel/Fonts -I/home/user/CLionProjects/Root/Kernel/Graphics -I/home/user/CLionProjects/Root/Kernel -g -std=c++20 -fcolor-diagnostics -g -mcmodel=large -Wall -Werror -Wextra -Wpedantic -ffreestanding -fno-stack-protector -fno-rtti -fno-exceptions -fno-pie -O0 -MD -MT CMakeFiles/Root.dir/src/Kernel.cpp.o -MF CMakeFiles/Root.dir/src/Kernel.cpp.o.d -o CMakeFiles/Root.dir/src/Kernel.cpp.o -c /home/user/CLionProjects/Root/src/Kernel.cpp
/home/user/CLionProjects/Root/src/Kernel.cpp:1:8: fatal error: module 'Kernel.Graphics' not found
import Kernel.Graphics;
~~~~~~~^~~~~~
1 error generated.
ninja: build stopped: subcommand failed.

I’m using cmake version 3.28, clang++ version 16.0.6 and ninja version 1.11.1.

Thanks in advance for your help,
Johnny

ben.boeckel · October 22, 2023, 4:09pm

Taking notes as I see them:

If you export module modname:partition;, you must export import :partition; in the primary unit. Given that you do not, I think you want just module PdSTL.Variant:VariantStorage; here (same for :VariantChoice).

Clang 17 is fine now; it was just Clang 16.

Do you also have headers? The directory tree seems truncated. If not, this doesn’t seem useful?

I see a Kernel.Graphics.Framebuffer module, but not a Kernel.Graphics module.

nitroglyceryne · October 22, 2023, 6:09pm

You are correct. The idea is that VariantChoice and VariantStorage shouldn’t be used by any user individually. They are needed for the Variant class only.

Ben Boeckel (Kitware):

Clang 17 is fine now; it was just Clang 16.

Clang 17 isn’t available in distribution’s repository. If you think the Clang version is the culprit, I can try building Clang 17 locally or check AUR.

I don’t have any headers. I wasn’t sure if this was needed for discoverability.

You’re correct. I wrote a wrong name in main.cpp.

I implemented changes per your review:

Changed export module PdSTL.Variant:VariantStorage to module PdSTL.Variant:VariantStorage
(Same for VariantChoice)
Removed target_include_directories()
Changed import Kernel.Graphics to import Kernel.Graphics.FrameBuffer

The error stays the same:

[1/199] Building CXX object CMakeFiles/Root.dir/src/Kernel.cpp.o
FAILED: CMakeFiles/Root.dir/src/Kernel.cpp.o 
/usr/bin/clang++  -I/home/user/CLionProjects/Root/PdSTL -I/home/user/CLionProjects/Root/Kernel/ASM -I/home/user/CLionProjects/Root/Kernel/Base -I/home/user/CLionProjects/Root/Kernel/BootInfo -I/home/user/CLionProjects/Root/Kernel/Debug -I/home/user/CLionProjects/Root/Kernel/EFI_Memory -I/home/user/CLionProjects/Root/Kernel/Fonts -I/home/user/CLionProjects/Root/Kernel/Graphics -I/home/user/CLionProjects/Root/Kernel -g -std=c++20 -fcolor-diagnostics -g -mcmodel=large -Wall -Werror -Wextra -Wpedantic -ffreestanding -fno-stack-protector -fno-rtti -fno-exceptions -fno-pie -O0 -MD -MT CMakeFiles/Root.dir/src/Kernel.cpp.o -MF CMakeFiles/Root.dir/src/Kernel.cpp.o.d -o CMakeFiles/Root.dir/src/Kernel.cpp.o -c /home/user/CLionProjects/Root/src/Kernel.cpp
/home/user/CLionProjects/Root/src/Kernel.cpp:1:8: fatal error: module 'Kernel.Graphics.FrameBuffer' not found
import Kernel.Graphics.FrameBuffer;
~~~~~~~^~~~~~
1 error generated.
ninja: build stopped: subcommand failed.

ben.boeckel · October 22, 2023, 7:06pm

Ok. Without a way to try this myself (remaking all of this out would be tedious and prone to introduce new errors or fix something unintentionally), the places to look (per target):

CXXDependInfo.json files (inter-target dependency information for the collator)
various .ddi files (output of scanning)
CXX.dd files (collator → ninja edge additions)
CXXModules.json for CMake-oriented module dependency information

If something seems missing or incorrect in there, that gives the next direction. If everything there looks fine, a Gist with a full tree of this example would be appreciated.

nitroglyceryne · October 24, 2023, 7:28pm

Hey, I looked at this again.

I made a pretty silly mistake in my local files.
Originally I intended FrameBuffer to be a module partition unit of module Kernel.Graphics. I wanted to add more partitions later.

In my local files Root/Kernel/Graphics/FrameBuffer.cppm had a line

export module Kernel.Graphics:FrameBuffer;

which caused the error.

When I was writing this post I didn’t copy & paste any file for the sake of brevity. I wrote what I believed to be important parts which is why the original post states:

which isn’t what I had locally.

Thanks for your help