Multiple conflicting paths for install RUNTIME_DEPENDENCY SET when library is not header-only on linux

I have a cmake project that creates a shared library that uses FFMPEG from vcpkg, and installs the shared library along with FFMPEG .so (or .dll) files using the RUNTIME_DEPENDENCY_SET option. On Windows, this works fine. However, on Linux, the install step fails with the following error message because I have multiple versions of ffmpeg on my system:

CMake Error at build/cmake_install.cmake:100 (file):
file Multiple conflicting paths found for libavutil.so.56:
/home/wanglab/Programs/vcpkg/installed/x64-linux-dynamic/lib/libavutil.so.56
/home/wanglab/mambaforge3/envs/moose/lib/./libavutil.so.56
/home/wanglab/mambaforge3/envs/moose/lib/libavutil.so.56

The problem only seems to exist when I return a type defined in ffmpeg, but not for functions where I use ffmpeg functions and types internally. Additionally, if I define my functions that return ffmpeg types in the header using inline, I am able to install the ffmpeg libraries. So this function is okay:

void say_hello(std::string codec_name)
{  
   ::AVCodec* codec = ::avcodec_find_encoder_by_name(codec_name.c_str());
    std::cout << "Hello World" << std::endl;
    std::cout << codec << std::endl;
}

but this function will produce the error unless i define it inline in the header file:

::AVDictionary* make_my_dict(const std::multimap<std::string, std::string> dict)
{
    ::AVDictionary* d = nullptr;
    for (const auto& entry : dict) {
        ::av_dict_set(&d, entry.first.c_str(), entry.second.c_str(), 0);
    }
    return d;
}

I find ffmpeg during configuration like this in my CMakeLists.txt:

find_package(PkgConfig REQUIRED)

pkg_check_modules(FFMPEG REQUIRED IMPORTED_TARGET
libavcodec
libavformat
libavutil
libswscale
libswresample)
target_link_libraries(ffmpeg_test PRIVATE PkgConfig::FFMPEG)

and the error only comes during installation with these commands:

install(TARGETS ffmpeg_test RUNTIME_DEPENDENCY_SET appDeps)
install(RUNTIME_DEPENDENCY_SET appDeps)

So there are no issues if I don’t try to copy the .so libraries into the installation folder.

I have been unsuccessful at creating a post_include_regex to select the right ffmpeg libraries in the RUNTIME_DEPENDENCY_SET block. I am able to select the correct (vcpkg) ffmpeg in the configuration step, and my cmakecache appears to be accurate. I have read that these variables do not propagate to the install commands. On linux, is there a way to prevent cmake from finding conflicting paths when multiple c libraries are in the path during install? Is there a reason this doesn’t happen if the troublesome functions are defined in the header?

I am using CMake 3.23.1. Please let me know if you need additional information.

It seems to me like you should avoid having Anaconda (or whatever flavor you’re using) and vcpkg at the same time. Both want to control LD_LIBRARY_PATH which means that any package installed in both (when loaded) are bound to conflict.

Thank you for your reply. Interestingly, if I deactivate anaconda entirely, the same error message persists. In case anaconda was just being really sticky, I switched to a Docker image for my build without any anaconda. I still get an error which is slightly different:

CMake Error at build/cmake_install.cmake:106 (file):
  file Could not resolve runtime dependencies:

    libavcodec.so.58
    libavutil.so.56
    libswresample.so.3

I get the same error message on another linux machine without anaconda.

Note that during configure, and build steps, those libraries are found without errors.

The error only happens with the cmake install command, and I am able to run the program if I don’t use the RUNTIME_DEPENDENCY_SET install command and instead manually copy the libraries into the same directory. If I objdump the shared library that I created, it has the location of the vcpkg ffmpeg libraries in its runpath:

# objdump -x libffmpeg_wrapper.so | grep vcpkg
  RUNPATH              /vcpkg/installed/x64-linux-dynamic/lib:

There is also this weird component where if I inline everything that calls ffmpeg, the install step works. However, it seems that I need to inline functions that call functions that call ffmpeg, so this isn’t such a simple fix. I could explore making a header only INTERFACE library for any ffmpeg function calls I might need and see if this changes anything. Currently, I have PRIVATE include directories for my ffmpeg function calls.

On linux, what programs is the RUNTIME_DEPENDENCY step using to find what libraries to install? FFMEPG is a c library, so I have to put the includes in a extern “C” block; is it possible there is some name mangling issue that is confusing whatever linux program has to find the libraries during install?

Did you install ffmpeg in this container?

Yes, ldd will just pick “the first” it finds. RUNTIME_DEPENDENCY_SET instead prefers to say “hey, you have something ambiguous, you should look at this”.

We read the ELF files and perform the search ourselves. ldd cannot be made to ignore things like LD_LIBRARY_PATH which would interfere with the intended behaviors.

Yes, I configured and compiled with the ffmpeg provided by vcpkg with the CMAKE_TOOLCHAIN_FILE variable pointing to my vcpkg.cmake file. If I install another ffmpeg in the container through apt-get, the error message becomes the following at the install step:

CMake Error at build/cmake_install.cmake:106 (file):
  file Multiple conflicting paths found for libavcodec.so.58:

    /lib/x86_64-linux-gnu/libavcodec.so.58
    /vcpkg/installed/x64-linux-dynamic/lib/libavcodec.so.58

This is the same as when another ffmpeg existed from anaconda (although I can understand this situation more because it would be in the path).

I am able to prevent the installation of system libraries with the PRE_EXCLUDE_REGEXES flag. I have tried selecting for one of the ffmpeg paths with the POST_INCLUDE_REGEXES flag such as

POST_INCLUDE_REGEXES ".*vcpkg.*"

to catch the <mypath_prefix>/vcpkg/<mypath_postfix> part of the path but I have had no luck that way.

So installing it on the system makes it go from “cannot find at all” to “I found two copies”? Some step isn’t being mentioned here.

This is my CMakeLists.txt:

cmake_minimum_required(VERSION 3.21)
project(ffmpeg_wrapper VERSION 0.1.0)

list(INSERT CMAKE_MODULE_PATH 0 ${CMAKE_SOURCE_DIR}/cmake)

set(CMAKE_INCLUDE_CURRENT_DIR ON)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

include(CTest)
enable_testing()

set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)

#Create Library
set(Sources
        src/videodecoder.cpp
        src/libavinc.cpp
        ffmpeg_wrapper.cpp
        src/videoencoder.cpp
)

set(Headers
    headers/videodecoder.h
    headers/libavinc.hpp
    headers/videoencoder.h
)

set(Public_Headers
    headers/videodecoder.h
    headers/videoencoder.h
)

add_library(ffmpeg_wrapper SHARED ${Sources} ${Headers})

#https://github.com/microsoft/vcpkg/pull/19308
find_package(PkgConfig REQUIRED)
pkg_check_modules(FFMPEG REQUIRED IMPORTED_TARGET
                libavcodec
                libavformat
                libavutil
                libswscale
                libswresample
                )

#add the include directories
target_include_directories(ffmpeg_wrapper
                           PRIVATE
                           #These are my directories
                           ${CMAKE_CURRENT_SOURCE_DIR}/headers
                           
                           PUBLIC
                           ${FFMPEG_INCLUDE_DIRS}
                           "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>"
                           "$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>"
)

target_link_libraries(ffmpeg_wrapper PRIVATE PkgConfig::FFMPEG)

# note that ${public_headers} has to be in quotes
set_target_properties(ffmpeg_wrapper PROPERTIES PUBLIC_HEADER "${Public_Headers}")

#install and create export set
# install the target and create export-set
install(TARGETS ffmpeg_wrapper EXPORT ffmpeg_wrapperTargets
        LIBRARY DESTINATION lib
        ARCHIVE DESTINATION lib
        RUNTIME DESTINATION bin
        INCLUDES DESTINATION include
        PUBLIC_HEADER DESTINATION include/ffmpeg_wrapper # include/SomeLibrary
)

install(TARGETS ffmpeg_wrapper RUNTIME_DEPENDENCY_SET appDeps)

IF (WIN32)

install(RUNTIME_DEPENDENCY_SET appDeps
        PRE_EXCLUDE_REGEXES
                [[api-ms-win-.*]] [[ext-ms-.*]] [[kernel32\.dll]] 
                [[bcrypt.dll]] [[mfplat.dll]] [[msvcrt.dll]] [[ole32.dll]] [[secur32.dll]] [[user32.dll]] [[vcruntime140.dll]]
                [[ws2_32.dll]]
                [[libgcc_s_seh-1\.dll]] [[libstdc\+\+\-6.dll]] 
        POST_EXCLUDE_REGEXES
                [[.*/system32/.*\.dll]]
)

ELSE()

install(RUNTIME_DEPENDENCY_SET appDeps
        PRE_EXCLUDE_REGEXES
                [[libc\.so\..*]] [[libgcc_s\.so\..*]] [[libm\.so\..*]] [[libstdc\+\+\.so\..*]]
                [[ld.*]] [[libbz2.*]] [[libdl.*]] [[libgmp.*]] [[libgnutls.*]] [[libhogweed.*]]
                [[libpthread.*]] [[librt.*]] [[libz.*]]
        POST_EXCLUDE_REGEXES
)

ENDIF()



#Install the header files
install(FILES ${Headers} DESTINATION include ${CMAKE_INSTALL_INCLUDEDIR})

target_link_options(ffmpeg_wrapper PRIVATE "-Wl,--as-needed")

set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)

include(CMakePackageConfigHelpers)

#Create config File
configure_package_config_file(${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in
  "${CMAKE_CURRENT_BINARY_DIR}/ffmpeg_wrapperConfig.cmake"
  INSTALL_DESTINATION cmake
)

#install config file
install(FILES
          "${CMAKE_CURRENT_BINARY_DIR}/ffmpeg_wrapperConfig.cmake"
        DESTINATION cmake
)

# generate and install export file
install(EXPORT ffmpeg_wrapperTargets
        FILE ffmpeg_wrapperTargets.cmake
        NAMESPACE ffmpeg_wrapper::
        DESTINATION cmake
)

#Generate the export targets for the build tree
export(EXPORT "ffmpeg_wrapperTargets"
    FILE "${CMAKE_CURRENT_BINARY_DIR}/cmake/ffmpeg_wrapperTargets.cmake"
    NAMESPACE ffmpeg_wrapper::
)

I am executing cmake configuration, build, and install with these commands (only install fails):

cmake --no-warn-unused-cli -DCMAKE_TOOLCHAIN_FILE:STRING=/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DVCPKG_TARGET_TRIPLET:STRING=x64-linux-dynamic -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE \
-DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_C_COMPILER:FILEPATH=/usr/bin/gcc \
-DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/g++ \
-S/ffmpeg_wrapper -B/ffmpeg_wrapper/build -G "Unix Makefiles"

cmake --build /ffmpeg_wrapper/build --config Release --target all -j 14 --

cmake --install /ffmpeg_wrapper/build --prefix /test