CMAKE_HAVE_LIBC_PTHREAD failed

Hi,

I want to build Pytorch which uses cmake for its building procedure. Since the system gcc is 4.8.5, I want to use a custom path installed gcc-6.1.0. Currently, when I run

$ export CC=/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc
$ export CXX=/storage/users/mnaderan/tools/gcc-6.1.0/bin/g++

I get this error:

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp

Run Build Command(s):/bin/gmake cmTC_bceed/fast && gmake: Warning: File `Makefile' has modification time 19 s in the future
/bin/gmake -f CMakeFiles/cmTC_bceed.dir/build.make CMakeFiles/cmTC_bceed.dir/build
gmake[1]: Entering directory `/storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp'
gmake[1]: Warning: File `CMakeFiles/cmTC_bceed.dir/flags.make' has modification time 19 s in the future
Building C object CMakeFiles/cmTC_bceed.dir/src.c.o
/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc   -DCMAKE_HAVE_LIBC_PTHREAD -fPIE   -o CMakeFiles/cmTC_bceed.dir/src.c.o   -c /storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_bceed
/storage/users/mnaderan/tools/cmake-3.15.4/bin/cmake -E cmake_link_script CMakeFiles/cmTC_bceed.dir/link.txt --verbose=1
/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc  -DCMAKE_HAVE_LIBC_PTHREAD   -rdynamic  CMakeFiles/cmTC_bceed.dir/src.c.o  -o cmTC_bceed
CMakeFiles/cmTC_bceed.dir/src.c.o: In function `main':
src.c:(.text+0x2f): undefined reference to `pthread_create'
src.c:(.text+0x3b): undefined reference to `pthread_detach'
src.c:(.text+0x4c): undefined reference to `pthread_join'
src.c:(.text+0x60): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status

In spite that I see cmake has detected gcc-6.1.0 correctly

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/g++

The CXX compiler identification is GNU, found in "/storage/users/mnaderan/pt/pytorch/build/CMakeFiles/3.15.4/CompilerIdCXX/a.out"

Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc

Configured with: ./configure --prefix=/storage/users/mnaderan/tools/gcc-6.1.0 --disable-multilib --enable-languages=c,c++
Thread model: posix
gcc version 6.1.0 (GCC)

Any idea on how to fix that for cmake? I didn’t get any feedback from Pytorch.

The quoted errors look like they come from CMakeFiles/CMakeError.log. That is a log of checks that failed but it doesn’t mean that the configure or generate steps failed to produce a build system. It is expected that some checks fail in normal operation. In this case perhaps the C runtime library really doesn’t have pthread symbols built into it, and that is common.

Does the actual console output from CMake show any errors?

The terminal output is

$ Numa_INCLUDE_DIR=/storage/users/mnaderan/tools/numa-dev/include/ Numa_LIBRARIES=/storage/users/mnaderan/tools/numa-dev/lib64/ python setup.py install --user
Building wheel torch-1.4.0a0+f2a35db
-- Building version 1.4.0a0+f2a35db
cmake -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/storage/users/mnaderan/pt/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python2.7/site-packages -DCUDNN_INCLUDE_DIR=/storage/users/mnaderan/cudnn/include/ -DCUDNN_LIBRARY=/storage/users/mnaderan/cudnn/lib/ -DNUMPY_INCLUDE_DIR=/storage/users/mnaderan/.local/lib/python2.7/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/bin/python -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_LIBRARY=/usr/lib64/libpython2.7.so.1.0 -DTORCH_BUILD_VERSION=1.4.0a0+f2a35db -DUSE_CUDA=True -DUSE_NUMPY=True /storage/users/mnaderan/pt/pytorch
-- The CXX compiler identification is GNU 6.1.0
-- The C compiler identification is GNU 6.1.0
-- Check for working CXX compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/g++
-- Check for working CXX compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc
-- Check for working C compiler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Not forcing any particular BLAS to be found
-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE - Failed
-- NUMA is not available
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
-- Turning off deprecation warning due to glog.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/storage/users/mnaderan/pt/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- MKL_THREADING = OMP
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- MKL_THREADING = OMP
CMake Warning at cmake/Dependencies.cmake:141 (message):
  MKL could not be found.  Defaulting to Eigen
Call Stack (most recent call first):
  CMakeLists.txt:378 (include)


CMake Warning at cmake/Dependencies.cmake:160 (message):
  Preferred BLAS (MKL) cannot be found, now searching for a general BLAS
  library
Call Stack (most recent call first):
  CMakeLists.txt:378 (include)


-- MKL_THREADING = OMP
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_intel - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_gf - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_intel - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_gf - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_intel - mkl_intel_thread - mkl_core - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_gf - mkl_intel_thread - mkl_core - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_sequential - mkl_core - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_sequential - mkl_core - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_sequential - mkl_core - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_sequential - mkl_core - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_core - gomp - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_core - iomp5 - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl_intel_lp64 - mkl_core - pthread - m - dl]
--   Library mkl_intel_lp64: not found
-- Checking for [mkl_intel - mkl_core - pthread - m - dl]
--   Library mkl_intel: not found
-- Checking for [mkl_gf_lp64 - mkl_core - pthread - m - dl]
--   Library mkl_gf_lp64: not found
-- Checking for [mkl_gf - mkl_core - pthread - m - dl]
--   Library mkl_gf: not found
-- Checking for [mkl - guide - pthread - m]
--   Library mkl: not found
-- MKL library not found
-- Checking for [Accelerate]
--   Library Accelerate: BLAS_Accelerate_LIBRARY-NOTFOUND
-- Checking for [vecLib]
--   Library vecLib: BLAS_vecLib_LIBRARY-NOTFOUND
-- Checking for [openblas]
--   Library openblas: /usr/lib64/libopenblas.so
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (open).
-- The ASM compiler identification is GNU
-- Found assembler: /storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Brace yourself, we are building NNPACK
-- Performing Test NNPACK_ARCH_IS_X86_32
-- Performing Test NNPACK_ARCH_IS_X86_32 - Failed
-- Found PythonInterp: /bin/python (found version "2.7.5")
-- NNPACK backend is x86-64
-- Failed to find LLVM FileCheck
-- Found Git: /bin/git (found version "1.8.3.1")
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- compiled but failed to run
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- compiled but failed to run
CMake Error at third_party/benchmark/CMakeLists.txt:231 (message):
  Failed to determine the source files for the regular expression backend


-- Configuring incomplete, errors occurred!
See also "/storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 751, in <module>
    build_deps()
  File "setup.py", line 310, in build_deps
    cmake=cmake)
  File "/storage/users/mnaderan/pt/pytorch/tools/build_pytorch_libs.py", line 56, in build_caffe2
    rerun_cmake)
  File "/storage/users/mnaderan/pt/pytorch/tools/setup_helpers/cmake.py", line 318, in generate
    self.run(args, env=my_env)
  File "/storage/users/mnaderan/pt/pytorch/tools/setup_helpers/cmake.py", line 142, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/storage/users/mnaderan/pt/pytorch/torch', '-DCMAKE_PREFIX_PATH=/usr/lib/python2.7/site-packages', '-DCUDNN_INCLUDE_DIR=/storage/users/mnaderan/cudnn/include/', '-DCUDNN_LIBRARY=/storage/users/mnaderan/cudnn/lib/', '-DNUMPY_INCLUDE_DIR=/storage/users/mnaderan/.local/lib/python2.7/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/bin/python', '-DPYTHON_INCLUDE_DIR=/usr/include/python2.7', '-DPYTHON_LIBRARY=/usr/lib64/libpython2.7.so.1.0', '-DTORCH_BUILD_VERSION=1.4.0a0+f2a35db', '-DUSE_CUDA=True', '-DUSE_NUMPY=True', '/storage/users/mnaderan/pt/pytorch']' returned non-zero exit status 1

At this line

CMake Error at third_party/benchmark/CMakeLists.txt:231 (message):
  Failed to determine the source files for the regular expression backend

I opened that file and went to that line and saw

# C++ feature checks
# Determine the correct regular expression engine to use
cxx_feature_check(STD_REGEX)
cxx_feature_check(GNU_POSIX_REGEX)
cxx_feature_check(POSIX_REGEX)
if(NOT HAVE_STD_REGEX AND NOT HAVE_GNU_POSIX_REGEX AND NOT HAVE_POSIX_REGEX)
  message(FATAL_ERROR "Failed to determine the source files for the regular expression backend")
endif()

The error occurs because none of the calls to cxx_feature_check succeeded. That function is not provided by CMake and so must be part of Pytorch. I suggest asking for help from Pytorch folks to use their project.

I asked but it seems this not directly related to pytorch.
I think although I have specified CC and CXX, I haven’t specified some paths for the pthread library. So an env variable or a cmake -D option may help.
I don’t maybe some thing is missed during gcc-6.1 compilation since I said I installed that from source.

Ah, the error is in third_party/benchmark/CMakeLists.txt. That comes from the google/benchmark project. The cxx_feature_check function is defined here and does not log the output of the check so it is not clear what goes wrong. One could open an issue in the google/benchmark project to point out this problem and ask for help diagnosing the check failures.

Thank you. I will check with them.

I am investigating this because of the following issue:

Like said there, I suspect it could be because of -lpthreads (the s at the end would be the problem).

But I don’t know where the test come from… is it part of CMAKE?

It looks like the -lpthreads was not really blocking my compilation of wyrsun.

I was able to push compilation more with:
sudo apt-get install libtolua++5.1-dev