2

I am trying to scp a specific file from a remote server to my local machine using Paramiko in Python 3.

Background: There is a directory mydir on the destination machine 198.18.2.2 that contains many timestamp directories that start with the name 2020...

Destination machine: 198.18.2.2

Source Machine: 198.18.1.1

So far I have managed to construct the command to be executed as follows -

cd "$(ls -1d /mydir/20* | tail -1)"; scp -o StrictHostKeyChecking=no email_summary.log root@198.18.1.1:/mydir/work/logs/email_summary_198.18.2.2.log

Code:

def remote_execute(dest_ip, cmd):
    """API to execute command on remote machine"""
    result = []
    sys.stderr = open('/dev/null')
    ssh_client = paramiko.SSHClient()
    ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    try:
        ssh_client.connect(dest_ip, username='root')
        stdin, stdout, stderr = ssh_client.exec_command(cmd)
        for line in stdout.readlines():
            result.append(line.strip())
        ssh_client.close()
        return result
    except paramiko.AuthenticationException:
        print("Authentication with the remote machine failed")
        return
    except paramiko.SSHException:
        print("Connection to remote machine failed")
        return
    except paramiko.BadHostKeyException:
        print("Bad host key exception for remote machine")
        return

Call: remote_execute('198.18.1.1', cmd)

The problem is ls -1d /mydir/20* | tail -1 always gives me the latest timestamp folder. But if the email_summary.log file is not present in that folder, I would like to look into next latest timestamp folder that has the file email_summary.log.

Essentially, scp the file from the latest timestamp folder that contains the file "email_summary.log". Can someone please help me with this?

Thanks in advance.

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
Ira
  • 547
  • 4
  • 13

2 Answers2

3

Executing scp command on the remote machine to push the file back to the local machine is an overkill. And in general relying on shell commands is very fragile approach. You better use native Python code only, to identify the latest remote file and pull it to your local machine. Your code will be way more robust and readable.

sftp = ssh.open_sftp()
sftp.chdir('/mydir')

files = sftp.listdir_attr()

dirs = [f for f in files if S_ISDIR(f.st_mode)]
dirs.sort(key = lambda d: d.st_mtime, reverse = True)

filename = 'email_summary.log'

for d in dirs:
    print('Checking ' + d.filename)
    try:
        path = d.filename + '/' + filename
        sftp.stat(path)
        print('File exists, downloading...')
        sftp.get(path, filename)
        break
    except IOError:
        print('File does not exist, will try the next folder')

The above is based on:

Side note: Do not use AutoAddPolicy. You lose security by doing so. See Paramiko "Unknown Server".

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
0

How about finding for file (not directory) using find?

find /mydir/20* -name email_summary.log | sort | tail -1

This will give you you a path to the latest file to copy.

So, your command will look like that:

scp -o StrictHostKeyChecking=no "$(find /mydir/20* -name email_summary.log | sort | tail -1)" root@198.18.1.1:/mydir/work/logs/email_summary_198.18.2.2.log
Tupteq
  • 2,986
  • 1
  • 21
  • 30
  • Thanks for the solution. But how can i combine it with the 'scp' command so that the output of 'find' can be consumed ? – Ira Feb 14 '20 at 03:31
  • Thanks again. Apparently, the command is correct but is not getting executed using paramiko. When i run the command as is from the remote machine, it executes just fine. Need to figure out why isn't it being executed through paramiko. – Ira Feb 15 '20 at 05:24