0

I am trying to setup CUDA enabled Python & TensorFlow environment on OSx 10.11.6

Everything went quite smoothly. First I installed following:

  • CUDA - 7.5
  • cuDNN - 5.1

I ensured that the LD_LIBRARY_PATH and CUDA_HOME are set properly by adding following into my ~/.bash_profile file:

export CUDA_HOME=/usr/local/cuda 
export DYLD_LIBRARY_PATH="$CUDA_HOME/lib:$DYLD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:$LD_LIBRARY_PATH"
export PATH="$CUDA_HOME/bin:$PATH"

Then I used Brew to install following:

  • python - 2.7.12_2
  • bazel - 0.3.2
  • protobuf - 3.1.0

Then I used Pip to install CPU only TensorFlow from: https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.11.0rc0-py2-none-any.whl

I checked out the Magenta project from: https://github.com/tensorflow/magenta and run all the test using:

bazel test //magenta/...

And all of them have passed.

So far so good. So I decided to give the GPU enabled version of TensorFlow a shot and installed it from: https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.11.0rc0-py2-none-any.whl

Now all the tests fail with the following error:

import tensorflow as tf
  File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 23, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
    _pywrap_tensorflow = swig_import_helper()
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so, 10): Library not loaded: @rpath/libcudart.7.5.dylib
  Referenced from: /usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so
  Reason: image not found

So obviously the script run from Bazel has trouble locating the libcudart.7.5.dylib library.

I did try running GPU computations from Python without Bazel and everything seems to be fine.

I also did create a test script and run it using Bazel and it seems that the directory containing libcudart.7.5.dylib library is reachable, however the LD_LIBRARY_PATH is not set.

I searched the documentation and found --action_env and --test_env flags, but none of them actually seems to set the LD_LIBRARY_PATH for the execution.

These are the options from loaded from .bazelrc files.

Inherited 'common' options: --isatty=1 --terminal_columns=80
Inherited 'build' options: --define=allow_oversize_protos=true --copt -funsigned-char -c opt --spawn_strategy=standalone
'run' options: --spawn_strategy=standalone

What is the correct way to let Bazel know about the runtime dependencies?

UPDATE

The trouble seems to be caused by the fact that "env" command is part of the execution chain and it does seem to clear both LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environmental variables. Is there a workaround different than disabling the SIP?

Binus
  • 1,065
  • 7
  • 14

3 Answers3

1

It looks like SIP affects the behavior of how the DYLD_LIBRARY_PATH gets propagated to the child processes. I found a similar problem and another similar problem.

I didn't want to turn the SIP off, so I just created symlinks for the CUDA library into a standard location.

ln -s /usr/local/cuda/lib/* /usr/local/lib

Not sure if this is the best solution, but it does work and it does not require the SIP to be disabled.

Binus
  • 1,065
  • 7
  • 14
0

Use 

export LD_LIBRARY_PATH=/usr/local/cuda/lib64/

before launching bazel. Double check in the directory above if there is such a file.

ls /usr/local/cuda/lib64/libcudart.7.5.dylib

Note that in Macosx the name is different:

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib/

See this answer for more information on SuperUser

Community
  • 1
  • 1
fabrizioM
  • 46,639
  • 15
  • 102
  • 119
  • Does it have to be lib64 directory? My CUDA comes with a lib directory only. I tried to rename it and export the LD_LIBRARY_PATH, but no luck. Again if I run python somefile.py everything is fine, but if I do bazel //directory:target_with_the_same_file it's broken. So I think I am missing some Bazel options and my environment setup is fine. – Binus Oct 18 '16 at 17:35
  • no it does not matter if is lib64 or lib, but it has to match the path. Are you on macosx ? the LD_LIBRARY_PATH is different on mac. is DYLD_LIBRARY_PATH – fabrizioM Oct 18 '16 at 18:28
  • Thanks for the suggestions. I actually set both the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH just to be sure. From the tests I run so far it seems that Python is able to pick either one of them, whereas the Bazel just clears them both out during execution. – Binus Oct 18 '16 at 19:22
  • Please take a look at my answer here. http://stackoverflow.com/a/41073045/1831325 – norman_h Dec 10 '16 at 07:34
0

The problem is indeed SIP, and the solution is to pass --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib to the bazel command, e.g.:

bazel build -c opt --config=cuda --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib //tensorflow/tools/pip_package:build_pip_package

Roee Shenberg
  • 1,457
  • 1
  • 13
  • 22
  • Thanks for the answer. Building your own TensorFlow is certainly an option. However I remember having some issues with compatibility of all the dependencies so I ended up using the pre-build one. – Binus Jan 15 '17 at 19:30