0

I have here a bunch of files on my workstation. I wrote a multiprocessing python3 script which runs pretty fine. My workstation is a tiny-pc but I have a huge 40 Thread server which I would like to utilize with this software. I know I can manually rsync the files to the server, then execute the script there and rsync them back. But programming and doing something by hand isn't fun ;-) Thus, I would like to have the local files on my tiny-pc transferred to the server, processes there with my script and the output (.csv and plots) should be transferred back to my workstation.

How can I do that?

I think paramiko would be the software to go. This is what I got:

import paramiko
import sysrsync
from datetime import date
import time

today = date.today()
today_dmy = today.strftime("%d.%m.%Y")
input_local = '/mnt/c/Users/user/Documents/input/' + today_dmy
output_local = '/mnt/c/Users/user/Documents/output' 
input_external = '/home/user/input/' + today_dmy
output_external = '/home/user/output/' + today_dmy
ip = '192.168.10.6'
key = '/home/user/.ssh/id_rsa'

sysrsync.run(source = input_local,
        destination = input_external, 
        destination_ssh = ip,
        options = ['-avz'],
        private_key = key)
            


ssh_client=paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
session = ssh_client.connect(hostname=ip, username='user', password='pwd') 

stdin, stdout, stderr = ssh_client.exec_command('python3 /home/user/Programm.py')
print(stderr.readlines())
print(stdout.readlines())

time.sleep(10)
sysrsync.run(source = output_external,
        destination = output_local,
        source_ssh = ip,
        options=['-avz'],
        private_key = key)

ssh_client.close()

The rsync-part is working, but I get the error that the modules which are loaded in "Programm.py" aren't available. However, on the server these modules ARE installed and working (if I login into the server and execute "python3 Programm.py" by hand it works). This is the error of "stderr"

['Traceback (most recent call last):\n', '  File "/home/user/Programm.py", line 8, in <module>\n', '    import pandas as pd\n', "ModuleNotFoundError: No module named 'pandas'\n"]

For me it seems that the code tries to execute the local python3 and not the one on the server. Or am I wrong?

What have I to do / What's wrong?

Nobody
  • 77
  • 7
  • See [Environment variable differences when using Paramiko](https://stackoverflow.com/q/31964108/850848). – Martin Prikryl Apr 08 '22 at 10:37
  • I set the folder into $PATH but it still doesn't work. Even get_pty doesn't work – Nobody Apr 08 '22 at 11:20
  • Shouldn't it be `PYTHONPATH`? https://stackoverflow.com/q/37233140/850848#65164141 – Martin Prikryl Apr 08 '22 at 11:24
  • I put it in the pythonpath as well. But still doesn't work. What do I need to put into stdin, stdout, stderr = ssh_client.exec_command()? – Nobody Apr 08 '22 at 11:35
  • Something like `"PYTHONPATH=\"$PYTHONPATH;/path/to/module\" && python3 /home/user/Programm.py"`. – Though the right way is to fix your startup scripts to set the environment correctly for all kinds of sessions. See [Some Unix commands fail with " not found", when executed using Python Paramiko exec_command](https://stackoverflow.com/q/55419330/850848). – Martin Prikryl Apr 08 '22 at 11:39
  • So like this? stdin, stdout, stderr = ssh_client.exec_command('PYTHONPATH="$PYTHONPATH:/home/user/anaconda3/lib/python3.9/site-packages/" && /home/user/Programm.py') <--- but this doesn't work. Still "no module named pandas" – Nobody Apr 08 '22 at 12:04
  • Maybe you should first ask separate pure-Python question on this (without Paramiko). Because I assume you will have the same problem, when you try `ssh user@hostname python3 /home/user/Programm.py`, right? So this is not really a Paramiko question, nor question about *"processing local files on a remote server"*. – Martin Prikryl Apr 08 '22 at 12:39

0 Answers0