Downloading files from SFTP server listed by remote command with Paramiko in Python

Question

I am using Paramiko to connect to the SFTP server from my local machine and download txt files from remote path. I am able to make successful connection and can also print the remote path and the files but i cannot get the files locally. I can print the file_path and file_name but not able to download all the files. Below is the code I am using:

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect(hostname=hostname, username=username, password=password, port=port)

remotepath = '/home/blahblah'
pattern = '"*.txt"'
stdin,stdout,stderr = ssh.exec_command("find {remotepath} -name {pattern}".format(remotepath=remotepath, pattern=pattern))
ftp = ssh.open_sftp()
for file_path in stdout.readlines():
   file_name = file_path.split('/')[-1]
   print(file_path)
   print(file_name)
   ftp.get(file_path, "/home/mylocalpath/{file_name}".format(file_name=file_name))

I can see the file_path and file_name like below from print statement but get error while using ftp.get for multiple files. I can copy a single file by hardcoding the name on source and destination.

file_path = '/home/blahblah/abc.txt'
file_name = 'abc.txt'
file_path = '/home/blahblah/def.txt'
file_name = 'def.txt'

I see one file is downloaded and then i get the following error:

FileNotFoundErrorTraceback (most recent call last)

Error trace:

Traceback (most recent call last):  
File "<stdin>", line 1, in <module>
File "...anaconda3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 769, in get
  with open(localpath, 'wb') as fl:
FileNotFoundError: [Errno 2] No such file or directory: 'localpath/abc.txt\n'

Unless you provide the text of the stack trace (not just its name or type), it's not obvious if it's the local file or the remote file that isn't found. If it's a local file that gives the error, then you probably need to create a directory first if that directory doesn't exist. — Charles Duffy, Jun 28 '18 at 18:00
That said, additional to the immediate issue this question is about, you have at least two additional potentially security-impacting bugs. The first of those is shell injection: Use `shlex.quote()` in Python 3 or `pipes.quote()` in Python 2 to get strings that are safe to substitute into `ssh.exec_command()`; otherwise, someone asking this program to retrieve files from `/tmp/$(rm -rf ~)` could cause you to have a really bad day. — Charles Duffy, Jun 28 '18 at 18:01
The second security-impacting bug is caused by the reliance on a newline-delimited stream to list filenames. If someone runs `d=/home/blahblah/$'\n'/etc/passwd$'\n'; mkdir "$d" && touch "$d/foo"`, then your `find` would return `/etc/passwd` as a result, and your code would copy it over despite not being under `/home/blahblah`. — Charles Duffy, Jun 28 '18 at 18:03
To fix that latter one, use `-print0` on the `find` command, and iterate over `stdout.read().split('\0')[:-1]` instead of using `stdout.readlines()`. There's probably a more efficient way to fix that too. — Charles Duffy, Jun 28 '18 at 18:04
You might also want to look at what the remote file *is* -- if it's a broken symlink, well, there's your error. (And really, you should be putting a type filter on the `find` to only look for types of things that `sftp.get()` will work with). — Charles Duffy, Jun 28 '18 at 18:10
As hinted by @CharlesDuffy already, finding files using `find` command, instead of using an SFTP API is indeed a bad solution and actually it is the immediate cause of the problems. See my answer. — Martin Prikryl, Jun 29 '18 at 06:01

Martin Prikryl · Accepted Answer · 2018-06-29T07:29:13.597

readlines does not remove newline from the line. So as you can see in the traceback, you are trying to create a file named abc.txt\n, what is not possible on many file systems, and mainly, it's not what you want.

Trim the trailing new lines from file_path:

for file_path in stdout.readlines():
    file_path = file_path.rstrip()
    file_name = file_path.split('/')[-1]
    # ...

Though you would have saved yourself lot of troubles, had you used a pure SFTP solution, instead of hacking it by executing a remote find command (what is a very fragile solution, as hinted in comments by @CharlesDuffy).

See List files on SFTP server matching wildcard in Python using Paramiko.

Side note: Do not use AutoAddPolicy. You lose security by doing so. See Paramiko "Unknown Server".

Downloading files from SFTP server listed by remote command with Paramiko in Python

1 Answers1