Kind of solved it with another hack. Idea is to move set(ENV{LD_LIBRARY_PATH} ...) to generated CTestTestfile.cmake.
set(ENV{LD_LIBRARY_PATH} ...)
CTestTestfile.cmake