0

I am getting the

"ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory"

error when I call my application with "sudo python3.6 ..." but not when I call just "python3.6 ..."

elastic beanstalk uses "sudo", so removing the command is not an option.

Only CUDA 9.0 is installed (which is the correct version for our environment), and /usr/local/cuda/lib64/libcublas.so.9.0 exists .

we created a separate ec2 instance with the same config and same AMI ID: CUDA9ClassicAmazonLinuxDLAMIwithMXNetTensorflowandCaffe2, and had no issues. It seems to be an issue present only with the elastic beanstalk.

$LD_LIBRARY_PATH = '/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:'

$PATH = '/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin'

any thoughts?

talonmies
  • 70,661
  • 34
  • 192
  • 269
Megan Hardy
  • 397
  • 4
  • 12
  • it's probably an environment issue. the `sudo` environment may not pick up your `LD_LIBRARY_PATH` variable. As a diagnostic, you could try: `sudo -E python3.6 ...`, see [here](https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo) – Robert Crovella Jul 24 '18 at 23:56
  • sudo -E python3.6 ... also gives the libcublas.so.9.0 error, we set os.environ['LD_LIBRARY_PATH'] within the python code as well, how would we go about setting the LD_LIBRARY_PATH so sudo can see it? – Megan Hardy Jul 25 '18 at 00:01
  • more diagnostic: 1. just to confirm, what is the output of `echo $LD_LIBRARY_PATH`, 2. what is the output of `sudo -E bash -c 'echo $LD_LIBRARY_PATH'` (you can edit your question with responses, if you wish) – Robert Crovella Jul 25 '18 at 00:07
  • I don't know if you were implying this, but `os.environ['LD_LIBRARY_PATH']` by itself would not be sufficient. You would need to do something like `os.environ['LD_LIBRARY_PATH']='/path/to/cudatoolkit/lib64'`, but probably that is what you meant. (If you did something like that before any import, I would have thought that would have fixed the issue as well.) – Robert Crovella Jul 25 '18 at 00:12
  • yes we are setting "os.environ['LD_LIBRARY_PATH'] = $path/to/libcublas.so.9.0" as shown above – Megan Hardy Jul 25 '18 at 00:14
  • There shouldn't be any $ in the path. It should be something like `os.environ['LD_LIBRARY_PATH']='/usr/local/cuda-9.0/lib64'` – Robert Crovella Jul 25 '18 at 00:17
  • there is no '$' – Megan Hardy Jul 25 '18 at 00:44
  • output for "sudo -E echo $LD_LIBRARY_PATH" == "echo $LD_LIBRARY_PATH" == "sudo echo $LD_LIBRARY_PATH", all as shown above – Megan Hardy Jul 25 '18 at 00:47
  • and yes we set the os.environ['LD_LIBRARY_PATH'] before any import (except os of course) – Megan Hardy Jul 25 '18 at 00:52

1 Answers1

4

Add a file for setting the path to the new library(CUDA in your case) in 

/etc/ld.so.conf.d/foo.conf

The contents of the file is the path to the CUDA library (/usr/local/cuda/lib64)

And call 

 sudo ldconfig

to make sure the library is included in the environment.

Sahil Shah
  • 96
  • 7