When working with a project that has 8 000 header files - I’ve stumbled upon a time it takes to install them them, when they’re were already installed (corresponding issue)
Performing installation for the first time takes 26 seconds. Reinstalling it, without any changes to the files or to destination directory, takes 3 minutes. But installing it again after, again without any changes, is very fast - just 2 seconds. And all reinstallations after this become very quick also.
mtime ninja -v install
# 26 s
mtime ninja -v install
# 180 s
mtime ninja -v install
# 2.292s
I’ve stumbled upon this issue before with other projects - especially the ones that install a lot of Python files. Though I haven’t investigated it before.
Now I was able to reproduce it with a simple example - need to create cmake file below and run the following snippet. You can change 500 to some bigger number to have more header files and make timing more dramatic.
mkdir build && cd build
# Create 500 .hxx files at `../inc`.
python -c "from pathlib import Path; cwd = Path.cwd().parent; include_dir = cwd / 'inc'; include_dir.mkdir(exist_ok=True); [(include_dir / f'{i}.hxx').write_text('#define test 42\n'*50) for i in range(500)]"
cmake .. -DCMAKE_INSTALL_PREFIX="install" -G Ninja
# `mtime` is just unix-`time` like utility
mtime ninja -v install
# -- Installing: headers-install/build/install/include/99.hxx
# 0.594 s
mtime ninja -v install
# -- Up-to-date: headers-install/build/install/include/99.hxx
# 1.583 s
mtime ninja -v install
# -- Up-to-date: headers-install/build/install/include/99.hxx
# 0.240s
CMakeLists.txt:
cmake_minimum_required(VERSION 3.21)
project(headers-install)
include(GNUInstallDirs)
file(GLOB INC_HEADERS "${CMAKE_CURRENT_SOURCE_DIR}/inc/*.hxx")
install(FILES ${INC_HEADERS} DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}")
I’ve been testing this on Windows only with cmake 4.2.0, haven’t tested this on Unix yet.
It looks like on the first run, since installation destination is not present, it just copies the file and everything is relatively fast.
On the second run, in theory it should compare timestamps (or what metadata cmake usually compares duing file(INSTALL)) and to avoid copying. But based on the time it takes - it’s first removing the files and then copies them, so this iteration becomes even longer than the first one. Though it’s showing Up-to-date:, so in theory it shouldn’t have copied them. Not sure about how it works exactly.
On the third run it’s indeed just compares timestamps and everything is very fast. I guess it’s either builds some kind of cache on the second run and is able to reuse it at the second run and proceeding runs. But then in theory it should have been built the cache after the first run too.
Any idea what causes this? Is this a bug?
