Building application using ExternalProject_Add on M1 CPU

Code
1

Hello there,

I encountered a problem recently where I get a series of link errors while attempting to build an application on an M1 (ARM) CPU (this is on a 2021 MacBook) when linking against libraries built using ExternalProject_Add. I have a ‘fix’, but I’d like to check if it’s sufficient or if there’s a better, more idiomatic way of handling it.

This is the application in question - sdl-bgfx-imgui-starter

It’s a little graphics demo starter that relies on three separate dependencies (SDL2, bgfx and Dear ImGui). I have third-party folder that contains a single CMakeLists.txt file that uses ExternalProject_Add to download, build and install all the libraries.

The main application then uses DCMAKE_PREFIX_PATH from the cli and find_package from the CMakeLists.txt file to find and link the libraries.

This all seems to work fine for the most part, but when building on M1 hardware, when the main application tries to link against the built libraries, I get an avalanche of link errors for the calls in the libraries I’m consuming.

... building for macOS-arm64 but attempting to link with file built for macOS-x86_64

"_SDL_Init", referenced from:
    _main in main.cpp.o
"bgfx::shutdown()", referenced from:
    _main in main.cpp.o
"ImGui::GetDrawData()", referenced from:
    main_loop(void*) in main.cpp.o

etc...

To ‘fix’ this, I found I can simply add this line at the top of the application CMakeLists.txt file (source).

if (APPLE)
  set(CMAKE_OSX_ARCHITECTURES  "x86_64")
endif ()

And then if I reconfigure and build, everything works fine. My concern is, is this the right thing to do?

It’s not clear to me why the libraries fetched using ExternalProject_Add are built for “x86_64” as opposed to “ARM” (there does not seem to be any mention of CMAKE_OSX_ARCHITECTURES in any of the dependencies CMakeLists.txt files). When building the application the default appears to be ARM which mismatches and causes all the link errors.

It would be nice to be able to specify which architecture the user of the application would like to build for as opposed to hardcoding it in the CMakeLists.txt file. I could then ensure the user chooses explicitly (at least on Apple/macOS).

I’d be very grateful if anyone could offer any advice/suggestions on how to cleanly handle this in a future proof and idiomatic way.

Thank you very much for your time!

Tom

2

Is there anything using Rosetta in the process tree? Usual suspects would be CMake itself and Ninja. Any process using Rosetta will launch the x86_64 variant of the binaries it invokes, so anything detecting “native” architectures will be flipped to x86_64 if the build tools don’t support arm64 natively.

3

Thanks for getting back to me @ben.boeckel,

Hmm that’s a good question… so actually thinking about it now, one difference I have between building the third-party folder and the main application, is I have a little .sh script to configure the main application so I don’t have to type out all the CMake commands every time. I wonder if this is causing the mismatch…

I’ll try the longhand way and see if that stops the error from happening and report back.

Thanks for the suggestion!

4

Unfortunately that hasn’t helped, but it could still be something along the lines of what you’re suggesting. If ExternalProject_Add under the hood is using Rosetta this could potentially be causing it?

What would be your suggestion for confirming this? Would using Activity Monitor reveal this kind of information?

Thanks!

5

I would write a tiny utility that is a fat binary with arm64 and x86_64 implementations that dumps out sysctlbyname("sysctl.proc_translated") (see its manpage for docs on usage) to tell whether it is running as x86_64 or arm64 (though the preprocessor may be simpler; that’s up you). Run this utility at various places to inspect the state at different points to try and probe where you’re missing an arm64 binary.

Activity Monitor might have this information, but I have no idea where to look.

6

Okay sounds like a plan, I’ll give that a try at some point later this week or next and report back when I have any more info.

I suspect there potentially might be something up with ExternalProject_Add that’s causing this but I don’t have enough information right now to be sure. I’ll see if I can zero in on something.

Thanks for the suggestions, much appreciated.

7

Hey there,

I finally got around to looking into this. I’ve been able to resolve the problem but there definitely seems to be something a little strange going on with ExternalProject_Add I don’t fully understand…

The crux of the problem seemed to be when I configured and built using the Ninja generator with ExternalProject_Add, the libraries built from this wound up being x86_64. I used lipo -info on Ninja and found it was x86_64.

❯ lipo -info /usr/local/bin/ninja
Non-fat file: /usr/local/bin/ninja is architecture: x86_64

The odd thing is when using Ninja to build a simple application (not using ExternalProject_Add) the build artefacts would be arm64. This is where the strange mismatch was coming from for me.

# using Ninja generator

❯ lipo -info test # simple app
Non-fat file: test is architecture: arm64

❯ lipo -info lib/libimgui.cmake.a # lib built with ExternalProject_Add at the same time
Non-fat file: lib/libimgui.cmake.a is architecture: x86_64

The solution I found was to download the latest version of Ninja which is now a fat binary including both x86_64 and arm64 architectures. Now when I run configure and build using ExternalProject_Add the build artefacts are arm64, not x86_64 as they were before.

This subtle difference is still a bit of a puzzle to me, but it’s likely something that’s not going to impact too many people in future.

I also took your advice and made a little test program that used sysctl.proc_translated. This page was very helpful - about-the-rosetta-translation-environment.

I could then build and run the program like so…

❯ cmake -B build -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" -G Ninja
...
❯ cmake --build build
...

❯ arch -x86_64 build/test
translated

❯ arch -arm64 build/test
native

Where the code looked something like this…

int process_is_translated()
{
  int ret = 0;
  size_t size = sizeof(ret);
  if (sysctlbyname("sysctl.proc_translated", &ret, &size, NULL, 0) == -1) {
    if (errno == ENOENT) {
      return 0;
    }
    return -1;
  }
  return ret;
}

int main(int argc, char** argv)
{
  if (auto result = process_is_translated(); result == 0) {
    std::cout << "native\n";
  } else if (result == 1) {
    std::cout << "translated\n";
  } else {
    std::cout << "error\n";
  }
}

Thanks again for your help and I can now remove the hardcoded CMAKE_OSX_ARCHITECTURES option from my CMakeLists.txt file

8

This is because the direct one is made from a CMake using arm64, so the “native” detection is working before ninja ruins things. In the second case, ninja is running cmake, so it runs as x86_64, autodetects that arch and compiles for the “native” host.

9

Ah I see! Interesting :sweat_smile: Thanks for following up, it all makes a lot more sense now, much appreciated :+1: