Environmental Variables not setting during SSH session

When setting environmental variable in Toolchain file, I see it set when locally logged into build machine. When I run the same thing during a SSH session, it doesn’t set. Is this by design?

Example here: https://github.com/jwinarske/rpi-vision/blob/master/cmake/clang.toolchain.cmake