1

Using Python's paramiko I try to access a server I own remotely via an SSH connection. This server has a python script that I execute via my SSH connection (I actually execute a .sh file that calls this python script). The first line of this python script is:

print("Devices:", tf.config.experimental.list_physical_devices())

Here I only see:

Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

If I execute the same .sh file after opening the SSH connection with PuTTY that same line of code generates the following result:

Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Here it does show the GPU as available. I suppose this is related to the way I open the paramiko SSH connection. This is the code:

ssh = paramiko.SSHClient()  
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect(ip, port=port, username=u, password=pass)

cmd_to_execute = "./" + shFileName + ".sh"

ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(cmd_to_execute)
    
for line in ssh_stdout:
    print(line.strip('\n'))
for line in ssh_stderr:
    print('ERR: ' + line.strip('\n'))
    
ssh.close()

Why do I not have access to the GPU when the SSH connection is established using paramiko?

EDIT 1

If I allow tensorflow logs to be displayed, I get this error: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

And others that look more or less like this one but changing the .11.0

On the other hand, if I use PuTTY it loads the libraries correctly: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0

Bobbick
  • 255
  • 3
  • 10
  • 1
    What kind of server is it? What do you get if you do `ssh u@ip:port ./shFileName.sh`? (as a **one** line). It might be a similar issue to [Environment variable differences when using Paramiko](https://stackoverflow.com/q/31964108/850848). – Martin Prikryl Sep 16 '21 at 07:55
  • 1
    You should look at the output tensorflow prints in both cases, it will tell you what is missing to get GPU support, likely some library paths are not being setup. – Dr. Snoopy Sep 16 '21 at 08:17
  • @Dr.Snoopy you are right, I get this error "Could not load dynamic library 'libcudart.so.11.0'" when executing it from paramiko and this other message "Successfully opened dynamic library libcudart.so.11.0" when executing through putty. – Bobbick Sep 16 '21 at 10:05
  • You need to run the shell (bash, zsh, etc) when you connect through SSH, if you run the script directly the shell is not run and that is where some paths are setup (as in .bashrc). – Dr. Snoopy Sep 16 '21 at 10:08
  • @MartinPrikryl executing that command (actually `ssh u@ip -p port ./shFileName.sh`) I get the same error as executing it from python with paramiko (as you expected). I will check the link you provide. The server is a Ubuntu 18.04 server, I do not know what info you need about it. – Bobbick Sep 16 '21 at 11:45
  • 1
    Then you have exactly that problem. Your environment is missing the `LD_LIBRARY_PATH`. – Martin Prikryl Sep 16 '21 at 12:31

1 Answers1

1

Following the comments made by Martin Prikryl and DR. Snoopy I was able to solve the issue by modifying the startup scripts to set the same PATH for all sessions. I did this by modifying two files.

$HOME/.profile file original code had this part

if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi

and I changed it to:

# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi

$HOME/.bashrc original code had this part:

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

and I changed it to:

# If not running interactively, don't do anything
#case $- in
#    *i*) ;;
#      *) return;;
#esac

This way now the same libraries are loaded without taking into account which way I established the connection.

Bobbick
  • 255
  • 3
  • 10