Using python to ssh into multiple servers and grab the file with the same postfix

Question

I normally use a bash script to grab all the files onto local machine and use glob to process all the files. Just wondering what would be the best way to use python (instead of another bash script) to ssh into each server and process those files?

My current program runs as

 for filename in glob.glob('*-err.txt'):
        input_open = open (filename, 'rb')
        for line in input_open:
            do something

My files all have the ending -err.txt and the directories where they reside in the remote server have the same name /documents/err/. I am not able to install third party libraries as I don't have the permission.

UPDATE

I am trying to not to scp the files from the server but to read it on the remote server instead..

I want to use a local python script LOCALLY to read in files on remote server.

score 0 · Answer 1 · edited May 23 '17 at 12:33

0

The simplest way to do it is to use paramico_scp to use ssh copy from the remote server (How to scp in python?)

If you are not allowed to download any libraries, you can create SSH key pair so that connecting to server does not require a password (https://www.debian.org/devel/passwordlessssh). You then can for each file do

import os
os.system('scp user@host:/path/to/file/on/remote/machine /path/to/local/file')

Note that using system is usually considered less portable than using libraries. If you give the script that use system('scp ...') to copy the files and they do not have SSH key pair set up, they will experience problems

edited May 23 '17 at 12:33

Community

1
1

answered Aug 23 '16 at 12:54

Dmitry Torba

3,004
1
14
24

I am trying to not to download it from the server but read it on the remote server instead..is it possible? – user3669481 Aug 23 '16 at 13:11
It looks like @user3669481 doesn't know what the names of the files are locally. – Steven Shaw Aug 23 '16 at 13:11
@StevenShaw not sure what you meant by this. All the files have the ending ```-err.txt``` everything before ```-``` is server specific.. – user3669481 Aug 23 '16 at 13:13
I just mean that the above solution would not work as the glob needs to execute remotely. – Steven Shaw Aug 23 '16 at 13:16

score 0 · Answer 2 · edited May 23 '17 at 12:33

0

Looks like you want to use a local Python script remotely. This has been answered here.

edited May 23 '17 at 12:33

Community

1
1

answered Aug 23 '16 at 13:14

Steven Shaw

6,063
3
33
44

No. I want to use a local python script locally but reading files on remote server – user3669481 Aug 23 '16 at 13:15
Well, you keep changing your question :) – Steven Shaw Aug 23 '16 at 13:18
How do you expect to `open` a remote file locally? – Steven Shaw Aug 23 '16 at 13:19
You might want to try sftp http://docs.paramiko.org/en/2.0/api/sftp.html – Steven Shaw Aug 23 '16 at 13:21

Using python to ssh into multiple servers and grab the file with the same postfix

2 Answers2