How to read multiple files from multiple remote servers in Python?

Question

I have a list of AWS ubuntu servers e.g

ubuntu@ec2-bla-95-blablabla-23.amazonaws.com
ubuntu@ec2-bla-95-blablabla-43.amazonaws.com
ubuntu@ec2-bla-95-blablabla-24.amazonaws.com
...

On each of these servers, I have a folder with variable number of files, the path is the same for each server e.g /roth/files/

I want to write a Python script that would fetch the contents of those files and merge them locally on my machine.

How do I go about fetching contents of those files on remote servers?

The way I login to these servers is

ssh -i  path/aws.pem ubuntu@ec2-bla-95-blablabla-23.amazonaws.com

e.g using a key

I found answer on similar question here

sftp_client = ssh_client.open_sftp()
remote_file = sftp_client.open('remote_filename')
try:
    for line in remote_file:
        # process line
finally:
    remote_file.close()

But I do not see where you provide a server name, and the key...

EDIT: As a small correction to Ganesh's answer you need to do the following to fetch every file or otherwise you get an error complaining that you try to fetch directory:

lobj = sftp.listdir_attr(target_folder_remote)
    for o in lobj:
        name = o.filename
        sftp.get(os.path.join(target_folder_remote, name), os.path.join(target_folder_local, name))

Ganesh Kathiresan · Accepted Answer · 2019-11-09T04:31:09.613

3

aws_host_list = [] # Your list here

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
dest_list = list()
for host in aws_host_list:
    client.connect(host, username=<User Name>, key_filename=<.PEM File path)
    sftp = client.open_sftp()
    lobj = sftp.listdir_attr(target_folder_remote)
    for o in lobj:
        name = o.filename
        sftp.get(os.path.join(target_folder_remote, name), os.path.join(target_folder_local, name))
        dest_list.append(os.path.join(target_folder_local, name))
    sftp.close()
    client.close()

# Then process you files
for f in dest_list:
    # Combine it

edited Nov 09 '19 at 04:31

answered Nov 08 '19 at 17:07

Ganesh Kathiresan

2,068
2
22
33

I am a bit unclear what to provide in +host – YohanRoth Nov 08 '19 at 17:21
Also looking at docs it says that .put is used to "Copy a local file (localpath) to the SFTP server as remotepath". I want the reverse right... Copy remote to local – YohanRoth Nov 08 '19 at 17:30
I think it should be .get() instead of .put()? – YohanRoth Nov 08 '19 at 17:31
My bad yeah, I'll fix it – Ganesh Kathiresan Nov 08 '19 at 17:33
so when I use .get() on a directory it throws error: IsADirectoryError: [Errno 21] Is a directory: How do I fetch a list of files? I think I need to call get() on every file spearately – YohanRoth Nov 08 '19 at 17:56
I guess I also need to use listdir_attr and get filenames from it – YohanRoth Nov 08 '19 at 17:57
I updated question with partial correction to your answer. That seems to work. Maybe you can update your answer so I can accept it – YohanRoth Nov 08 '19 at 18:05

How to read multiple files from multiple remote servers in Python?

1 Answers1