4

I have a Conda environment with cuda, cudnn, cudatoolkit installed from the nvidia channel and tensorflow installed from PyPI.

When I try import tensorflow, I get

could not load dynamic library 'libcudart.so.11.0';

even though it is in $CONDA_PREFIX/lib.

I can work around this by appending/prepending this path to LD_LIBRARY_PATH manually (shared objects are in lib directory). I know this can be automated via a script in opt/conda/activate.d/ but shouldn't conda activate env-name resolve this automatically?

Is something wrong with my Conda installation / environment?

Configuration Details

Python version

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux

Relevant Conda Packages

$ conda list -n env-name '(cuda|tensorflow)'
# Name                    Version                   Build  Channel
cuda                      11.5.0               hc28fa2a_0    nvidia
cuda-cccl                 11.5.62                       0    nvidia
cuda-command-line-tools   11.5.0               hc28fa2a_0    nvidia
cuda-compiler             11.5.0               hc28fa2a_0    nvidia
cuda-cudart               11.5.50              h79feb7f_0    nvidia
cuda-cudart-dev           11.5.50              h2215e51_0    nvidia
cuda-cuobjdump            11.5.50              h3fb44dc_0    nvidia
cuda-cupti                11.5.57              h2e3e70b_0    nvidia
cuda-cuxxfilt             11.5.50              he5660e7_0    nvidia
cuda-driver-dev           11.5.50                       0    nvidia
cuda-gdb                  11.5.50              h03b1d8f_0    nvidia
cuda-libraries            11.5.0               hc28fa2a_0    nvidia
cuda-libraries-dev        11.5.0               hc28fa2a_0    nvidia
cuda-memcheck             11.5.50              h29df99e_0    nvidia
cuda-nvcc                 11.5.50              h8f81028_0    nvidia
cuda-nvdisasm             11.5.50              hc026f5c_0    nvidia
cuda-nvml-dev             11.5.50              h511b398_0    nvidia
cuda-nvprof               11.5.50              h1597bcb_0    nvidia
cuda-nvprune              11.5.50              h003a74f_0    nvidia
cuda-nvrtc                11.5.50              h895d16b_0    nvidia
cuda-nvrtc-dev            11.5.50              h2215e51_0    nvidia
cuda-nvtx                 11.5.50              h4855407_0    nvidia
cuda-nvvp                 11.5.50              h97cf613_0    nvidia
cuda-runtime              11.5.0               hc28fa2a_0    nvidia
cuda-samples              11.5.56              hf1e648b_0    nvidia
cuda-sanitizer-api        11.5.50              h7b563e8_0    nvidia
cuda-toolkit              11.5.0               hc28fa2a_0    nvidia
cuda-tools                11.5.0               hc28fa2a_0    nvidia
cuda-visual-tools         11.5.0               hc28fa2a_0    nvidia
cudatoolkit               11.5.0               h36ae40a_9    nvidia
tensorflow                2.7.0                    pypi_0    pypi
tensorflow-estimator      2.7.0                    pypi_0    pypi
tensorflow-hub            0.12.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.21.0                   pypi_0    pypi
merv
  • 67,214
  • 13
  • 180
  • 245
Shriraj Hegde
  • 829
  • 5
  • 18
  • Conda uses **@rpath** directives, not `LD_LIBRARY_PATH` to resolve environment-relative libraries. Please add output from `conda list -n env-name '(cuda|tensorflow)'` to get more specific help. – merv Nov 10 '21 at 17:40
  • @merv Output added to the question, Can you please take a look – Shriraj Hegde Nov 10 '21 at 18:31
  • 1
    Thanks for adding that. Okay, I was able to replicate this, but don't have time to troubleshoot. The fact that [a similar question](https://stackoverflow.com/q/69917132/570918) is popping up makes me suspect it could be something awry with the repository (not your fault). I'm also not so familiar with the `nvidia` channel and those packages are less than a month old. Were there specific directions somewhere that suggested using `nvidia` channel + PyPI TensorFlow? Maybe also report your Python version. – merv Nov 10 '21 at 23:08
  • No specific direction, I just used `nvidia` channel for CUDA related packages and I just used pypy for tensorflow since pip has worked well for me in the past. With python 3.10 on fedora 35, some packages seem to be broken, so I just used the official python way for tensorflow. – Shriraj Hegde Nov 11 '21 at 06:05
  • @merv can you please edit the question title to a more suitable wording? Since Conda does not use `LD_LIBRARY_PATH` question doesn't make very much sense. I wrote that since setting `LD_LIBRARY_PATH` solved the problem. – Shriraj Hegde Nov 11 '21 at 06:08
  • Does this github [reference](https://github.com/tensorflow/tensorflow/issues/45930#issuecomment-770342299) help you to resolve this issue? –  Nov 15 '21 at 09:21
  • Probably not since I am not using system-wide cuda and cudatoolkit installation. Cuda and cudatoolkit are installed in conda env – Shriraj Hegde Nov 16 '21 at 15:35
  • You need to follow [this](https://www.tensorflow.org/install/pip) document and verify that you have followed all the steps mentioned to install `Tensorflow` in [anaconda](https://docs.anaconda.com/anaconda/user-guide/tasks/tensorflow/) and to enable [GPU support](https://www.tensorflow.org/install/gpu) in your system. –  Apr 21 '22 at 15:48

0 Answers0