copy several files with identical filename from server to local and with python paramiko

Question

I would like to copy several files with specific, identical name from the server to local using paramiko (for a school project). However I would like to have a list of servers for the script to go through and execute the same code and also detect if the server is online or not. How do I do this?

There are several identical files that I do not need. I need to pull the specific "dblatmonstat" files only.

Example of file name pc_dblatmonstat_dpc01n1_scl000101014.log

Like: first go through...

dpc01n1.sccloud.xxx.com

And then the same code through... dpc02n1.sccloud.xxx.com

...and so on and so forth.

Here is what I have so far:

import os
import paramiko
import re

#Create log file
#paramiko.util.log_to_file('/$PMRootDir/SrcFiles/logfetcher.log')
#paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))

#Credentials
host = 'dpc01n1.sccloud.xxx.com'
port = 22
username = 'pi'
password = 'pi'

#Locations
files = re.search('?<=pc_dblatmonstat_dpc01n1_)\w+', files)
print('There are files:', files)
remote_path = '/home/pi/Desktop/logs'
local_path = r'C:\Users\urale\Desktop\logs'


#Opening ssh and ftp
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(
            paramiko.AutoAddPolicy())
ssh.connect(host, username, port, password)
sftp = ssh.open_sftp()

#Getting files
for file in files:
    file_remote = remote_path + file
    file_local = local_path + file

    print (file_remote) + '>>>' + file_local

    #sftp.get(file_remote, file_local)
    sftp.put(file_local, file_remote)

sftp.close()
ssh.close()

EDIT:

This version keeps downloading the same file over and over again. How can I break the loop when a file has been finished downloading and moves to the next server? I'm also trying to use the re.search function in order to only download the pc_dblatmonstat_xxxxxxxxxx_xxxxxxxxxxxxx.log files. re.search should match dblatmonstat__.log something like this...

import os
import paramiko
import re

# You could add the local_path to the function to define individual places for the 
# files that you download.
Lpath = 'C:\\'
Lpath1 = 'Users'
Lpath2 = 'urale'
Lpath3 = 'Desktop'
Lpath4 = 'logs\\'
local_path = os.path.join(Lpath, Lpath1, Lpath2, Lpath3, Lpath4)

Rpath1 = 'home'
Rpath2 = 'pi'
Rpath3 = 'Desktop'
Rpath4 = 'logs'
remote_path = os.path.join(Rpath1, Rpath2, Rpath3, Rpath4)

# 1. Create function
def get_server_files(local_path, host, port, username, password, remote_path, files):
    #Opening ssh and ftp
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(host, port, username, password)
    sftp = ssh.open_sftp()

    #Getting files
    for file in files:

        file_remote = remote_path + files
        file_local = local_path + files

        print(file_remote, '>>>', file_local)

        sftp.get(file_remote, file_local)
        #sftp.put(file_local, file_remote)

    sftp.close()
    ssh.close()

# 2. list of servers
# Add new dictionary for each server to this list
list_of_servers = [
    { 'host': '192.168.1.64',
      'port': 22, 
      'username': 'pi', 
      'password': 'pi', 
      'remote_path': '/home/pi/Desktop/logs/', 
      'files':  'pc_dblatmonstat_dpc01n1_scl000101014.log'
      }
]

# 3. Iterate through the list_of_servers, using the function above
for server in list_of_servers:
    get_server_files(local_path, **server)

I think you could benefit from using fabric: http://www.fabfile.org/. Then you will be able to control the various servers and also execute scripts on the servers. — Philip, Apr 04 '20 at 12:19
I forgot to mention that I only have read access to the servers. The script however should only connect to each server and pull the content of a specified path (like above) and store it in a certain directory. — Trunks, Apr 06 '20 at 14:26

Philip · Accepted Answer · 2020-04-08T04:34:03.093

I have not tested the following, but it should work and give you an idea of how to approach the problem.

Convert script to function
Create a list of servers
Iterate through the list with the function to retrieve the files

This is reflected in the following

import os
import paramiko
import re

# 1. Create function
def get_server_files(local_path, host, port, username, password, remote_path, file_pattern):
    """Connects to host and searches for files matching file_pattern
    in remote_path. Downloads all matches to 'local_path'"""
    #Opening ssh and ftp
    ssh_con = paramiko.SSHClient()
    ssh_con.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh_con.connect(host, port, username, password)
    sftp_con = ssh_con.open_sftp()

    # Finding files
    all_files_in_path = sftp_con.listdir(path=remote_path)
    r = re.compile(file_pattern)
    files = list(filter(r.match, all_files_in_path))

    #Download files
    for file in files:
        file_remote = remote_path + file
        file_local = local_path + file

        print(file_remote) + '>>>' + file_local

        sftp_con.get(file_remote, file_local)
        #sftp_con.put(file_local, file_remote)

sftp_con.close()
ssh_con.close()

# 2. list of servers
# Add new dictionary for each server to this list
list_of_servers = [
    { 'host': 'dpc01n1.sccloud.xxx.com',
      'port': 22, 
      'username': 'pi', 
      'password': 'pi', 
      'remote_path': '/home/pi/Desktop/logs', 
      'file_pattern': 'pc_dblatmonstat_dpc01n1'}
]

# You could add the local_path to the function to define individual places for the
# files that you download.
local_path = r'C:\Users\urale\Desktop\logs'

# 3. Iterate through the list_of_servers, using the function above
for server in list_of_servers:
    get_server_files(local_path, **server)

Very much appreciated, Philip. Will test it tonight and let you know. Thank you very much for your effort! — Trunks, Apr 06 '20 at 21:24
Glad to help. Remember to mark the answer as correct if it help, so others may benefit from it too. — Philip, Apr 07 '20 at 02:35
For some reason, I get an error [Errno 10109] getaddrinfo failed in ssh.connect(host, username, port, password) — Trunks, Apr 07 '20 at 13:18
According to this https://stackoverflow.com/questions/5301378/python-paramiko-getaddrinfo-error-when-trying-to-establish-an-ssh-connection, you just need to change the order of the connection properties. I've updated the answer to handle the error. — Philip, Apr 07 '20 at 13:41
I see, thanks. It works - that's positive. But it also doesn't. It keeps downloading the same file over and over again in a loop and I couldn't figure out how to break the loop after the download has been finished. I also changed your version a bit. I added the newer code on my post above. Any advice? — Trunks, Apr 07 '20 at 19:43
Generally speaking you should start a new question if you have additional problems, however, I've updated the answer to handle finding the correct files. Have a look here for more clarification: https://stackoverflow.com/questions/3640359/regular-expressions-search-in-list. Please note that I have changed the input from being `files`, to `file_pattern`. — Philip, Apr 08 '20 at 04:37

copy several files with identical filename from server to local and with python paramiko

1 Answers1