Help with linking snappy to leveldb (Windows) pay 40$

[Reload]

snappy - GitHub - google/snappy: A fast compressor/decompressor
leveldb - GitHub - google/leveldb: LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

snappy: cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_POSITION_INDEPENDENT_CODE=on -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=ON .
cmake --build .

link to leveldb:

build leveldb: cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCMAKE_POSITION_INDEPENDENT_CODE=on -DLEVELDB_BUILD_TESTS=off -DLEVELDB_BUILD_BENCHMARKS=off .
cmake --build .

building plyvel (setup.py):

extra_compile_args = ['-Wall', '-g', '-x', 'c++', '-std=c++11']

if platform.system() == 'Darwin':
    extra_compile_args += ['-stdlib=libc++']

ext_modules = [
    Extension(
        'plyvel._plyvel',
        sources=['plyvel/_plyvel.cpp', 'plyvel/comparator.cpp'],
        libraries=['leveldb'],
        include_dirs=['C:\Program Files (x86)\leveldb\include'],
        library_dirs=['C:\Program Files (x86)\leveldb\lib'],
        extra_compile_args=extra_compile_args,
    )
]

error: ImportError: DLL load failed while importing _plyvel: The specified module could not be found.

I’ve tried many options, but nothing has worked
I’ll pay $40 in bitcoin to anyone who can help solve the problem.

[ben.boeckel (Ben Boeckel (Kitware))]

So everything builds and the module just fails to load? Or the plyvel module fails to build?

[Reload]

everything is successfully building but when importing the builded plyevel an error occurs

[ben.boeckel (Ben Boeckel (Kitware))]

I would recommend getting the path(s) that cannot be found from Python. It seems you’re on Windows; I think Python 3.10 or 3.11 got better error messages. If that’s not possible, ProcMon can be used to sniff error states; you’ll need to set up some filters to make it useful.

[Reload]

I have python 3.10. I think the problem is minor. Do you have telegram? I can pay right away and provide more screenshots and info

[retif]

I tried to reproduce your problem, and got the same result, and it happens no matter where I put the DLLs (leveldb.dll and snappy.dll), even if their location is definitely part of the PATH.

And as it turns out, starting with Python 3.8(?) there is now a more strict check for DLL loading paths, so one has to explicitly “allow” paths of interest with os.add_dll_directory(), and that seems to make it work:

Python 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32

>>> import plyvel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\temp\some\plyvel\plyvel\__init__.py", line 6, in <module>
    from ._plyvel import (  # noqa
ImportError: DLL load failed while importing _plyvel: The specified module could not be found.

>>> import os
>>> os.add_dll_directory("e:/temp/some/plyvel")
<AddedDllDirectory('e:/temp/some/plyvel')>

>>> import plyvel
>>> db = plyvel.DB('e:/temp/testdb/', create_if_missing=True)
>>> os.listdir("e:/temp/testdb/")
['000003.log', 'CURRENT', 'LOCK', 'LOG', 'MANIFEST-000002']

[Reload]

That’s weird… It didn’t work for me

[retif]

Can’t see on your screenshot if libraries are in fact inside the path you are adding. But if it’s the same path, then it is weird indeed. Is the error absolutely the same as before?

Also, looking at your DLLs sizes, mine are different:

$ du -hs ./*.dll
260K    ./leveldb.dll
212K    ./snappy.dll

I’ve built Release configuration for both, and it was the latest commits from both their repositories. If you have used the same revisions and configuration, then it is strange that your leveldb.dll is more than 3 times bigger than mine.

I would also check that leveldb build actually did link to snappy, because it doesn’t fail it it is not found, just builds without it.

[Reload]

Yes, the error is exactly the same. I’m building leveldb 1.22 and snappy 1.1.9
As specified here - https://github.com/wbolster/plyvel/tree/main/scripts

Have you built with the same cmake parameters and under windows? Why don’t we go to telegram? It’ll be quicker and more convenient

[Reload]

Clearly something’s wrong whl size only 98kb, with only leveldb ~200kb. And how did you link scappy to leveldb?

[retif]

We can use personal messages here on the forum, I prefer not mix my identities
Also, this discussion might be of interest for others, so it’s worth keeping it public, at least until moderators would ask us to stop spamming.

I didn’t use those scripts, here are my steps:

$ cd /e/temp
$ mkdir some && cd $_

$ git clone --depth 1 git@github.com:google/snappy.git
$ git clone --depth 1 git@github.com:google/leveldb.git
$ git clone --depth 1 git@github.com:wbolster/plyvel.git

$ cd ./snappy
$ git rev-parse HEAD
27f34a580be4a3becf5f8c0cba13433f53c21337
$ mkdir build && cd build
$ cmake -G Ninja -DBUILD_SHARED_LIBS=1 -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=1 -DCMAKE_INSTALL_PREFIX="../install" -DSNAPPY_BUILD_TES
TS=0 -DSNAPPY_BUILD_BENCHMARKS=0 ..
$ cmake --build . --target install

$ cd ../../leveldb
$ git rev-parse HEAD
068d5ee1a3ac40dabd00d211d5013af44be55bea
$ mkdir build && cd build
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=1 -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DLEVELDB_BUILD_TESTS=0 -DLEVELDB_BUILD_BENCHMARKS=0 -DCMAKE_INS
TALL_PREFIX="../install" -DCMAKE_PREFIX_PATH="e:/temp/some/snappy/install" ..
$ cmake --build . --target install

$ cd ../../plyvel
$ git rev-parse HEAD
fc99b8ebdd1e93ff53138c98bc3f90d786158cad
$ make
$ python setup.py bdist_wheel
$ du -hs ./dist/plyvel-1.5.0-cp311-cp311-win_amd64.whl
100K

$ pip install ./dist/plyvel-1.5.0-cp311-cp311-win_amd64.whl

Linking snappy from leveldb project:

# ...

find_package(Snappy CONFIG REQUIRED)
if(Snappy_FOUND)
  message(STATUS "Found Snappy: ${Snappy_DIR}")
endif()

include(CheckLibraryExists)
check_library_exists(crc32c crc32c_value "" HAVE_CRC32C)
# ugly, but they don't set usual variables for libraries discovery
check_library_exists(snappy snappy_compress "${Snappy_DIR}/../.." HAVE_SNAPPY)
check_library_exists(zstd zstd_compress "" HAVE_ZSTD)
check_library_exists(tcmalloc malloc "" HAVE_TCMALLOC)

message(STATUS "HAVE_SNAPPY: ${HAVE_SNAPPY}")

# ...

if(HAVE_SNAPPY)
  target_link_libraries(leveldb Snappy::snappy)
endif(HAVE_SNAPPY)

# ...

Just in case, here’s setup.py modifications too:

extra_compile_args = ['-Wall', '-g', '-x', 'c++', '-std=c++11', '-Ie:/temp/some/leveldb/install/include']

if platform.system() == 'Darwin':
    extra_compile_args += ['-stdlib=libc++']

ext_modules = [
    Extension(
        'plyvel._plyvel',
        sources=['plyvel/_plyvel.cpp', 'plyvel/comparator.cpp'],
        library_dirs=["e:/temp/some/leveldb/install/lib"],
        libraries=['leveldb'],
        extra_compile_args=extra_compile_args
    )
]

[retif]

…forgot to mention, in case it isn’t obvious, the path added with os.add_dll_directory() still needs to be discoverable (be part of the PATH), as this function seems to only “allow” loading from there, I don’t think it actually loads anything. And the path on your screenshot doesn’t look like it is part of the PATH (unless you actually added it to PATH).

In my case I was launching Python from the same working folder where the DLLs are, so they were discoverable.

[Reload]

snappy - 1.1.9
leveldb - 1.22

cd snappy
cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_POSITION_INDEPENDENT_CODE=on -DSNAPPY_BUILD_TESTS=0 -DSNAPPY_BUILD_BENCHMARKS=0 -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=ON .
cmake --build . --target install

cd levelbd
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCMAKE_POSITION_INDEPENDENT_CODE=on -DLEVELDB_BUILD_TESTS=off -DLEVELDB_BUILD_BENCHMARKS=off .
cmake --build . --target install

Linking snappy: similarly ↑

leveldb.dll - 877kb
leveldb.lib - 82kb

scappy.dll - 199kb
scappy.lib - 181kb

plyenv setup.py

extra_compile_args = ['-Wall', '-g', '-x', 'c++', '-std=c++11', '-IC:\Program Files (x86)\leveldb\include']

if platform.system() == 'Darwin':
    extra_compile_args += ['-stdlib=libc++']

ext_modules = [
    Extension(
        'plyvel._plyvel',
        sources=['plyvel/_plyvel.cpp', 'plyvel/comparator.cpp'],
        libraries=['leveldb'],
        library_dirs=['C:\Program Files (x86)\leveldb\lib'],
        extra_compile_args=extra_compile_args,
    )
]
whl - 98kb

python code:
import os

os.add_dll_directory(r"C:\Users\xxx\PycharmProjects\xxx\libs")

import plyvel


class LevelDB:
    def __init__(self, path: str):
        self._path = path
        self._db = plyvel.DB(path)

    def get_db(self):
        with self._db.iterator() as it:
            for k, v in it:
                print(k, v)

    def get_value(self, key: str):
        return self._db.get(bytes(key, 'utf-8'))


leveldb = LevelDB("C:\\Users\\xxx\\Desktop\\leveldb\\test")
leveldb.get_db()

Warning - Cannot find reference 'DB' in '__init__.py' 

Error - Process finished with exit code -1073741819 (0xC0000005)

[Reload]

I tried to run from the working directory, also added to the PATH, the result is the same

[Reload]

– Building for: Visual Studio 17 2022
– Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.19045.
– The C compiler identification is MSVC 19.33.31630.0
– The CXX compiler identification is MSVC 19.33.31630.0

[retif]

Works for me:

# ...

leveldb = LevelDB("e:/temp/testdb")

vals = leveldb.get_db()
#print(vals)

# added put_value() implementation
leveldb.put_value(b"key", b"ololo")
x = leveldb.get_value("key")
print(x)
# b'ololo'

My toolchain:

• Visual Studio 2022 17.7.5
• MSVC 19.37.32825.0
• Ninja 1.11.0
• CMake 3.25.1

[Reload]

I don’t know what to do anymore…

[ben.boeckel (Ben Boeckel (Kitware))]

Getting ProcMon on the error detecting side of the system logs will help to determine what DLL is not being found.

[Reload]

[Reload]

I assume the problem is FILE LOCKED WITH ONLY READERS?