Recursive directory download with Paramiko?

Question

I want to download a directory with unknown contents recursively via SSH and have been trying Paramiko. I have seen several examples how to upload directories but none that covers recursive download.

I can list all items in a directory but haven't been able to find a way of knowing if the item is a file (to download) or a directory (to call recursively).

transport = paramiko.Transport((MY_IP, 22))
transport.connect(username=MY_NAME, password=MY_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)

file_list = sftp.listdir(path='/home/MY_HOME_DIR')
    for item in file_list:
        # Here is an item name... but is it a file or directory?
        print(item)
sftp.close()
transport.close()

So how do I know if an item is a file or if it is a directory?

Is using rsync an option? http://rsync.samba.org – Claes Mogren Nov 29 '11 at 11:47 — Claes Mogren, Nov 29 '11 at 11:47

westmark · Accepted Answer · 2012-01-02T10:09:03.940

12

from stat import S_ISDIR

def isdir(path):
  try:
    return S_ISDIR(sftp.stat(path).st_mode)
  except IOError:
    #Path does not exist, so by definition not a directory
    return False

...assuming sftp is an open Paramiko SFTP connection.

edited Jan 02 '12 at 10:09

answered Nov 29 '11 at 08:08

westmark

909
8
15

Can't test this now, but this seems to add the last missing piece into a complete answer. Moving accepted flag here. – Arahman Dec 19 '11 at 08:56

score 3 · Answer 2 · answered Apr 10 '13 at 17:12

An old question, but a solution I came up with that works quite well, it's a little bit sloppy (typecasting and slashes and all) - but it does work.

Note this uses fabric.api.local to make the directories in the destination.

def sftp_get_recursive(path, dest, sftp=sftp):
    item_list = sftp.listdir(path)
    dest = str(dest)

    if not os.path.isdir(dest):
        local("mkdir %s" % dest)

    for item in item_list:
        item = str(item)

        if is_directory(path + "/" + item, sftp):
            sftp_get_recursive(path + "/" + item, dest + "/" + item, sftp)
        else:
            sftp.get(path + "/" + item, dest + "/" + item)

This answer as per 2021 requires a few updates; I posted the updated answer. — Krzysztof Maliszewski, Oct 20 '21 at 07:02

Martin Prikryl · Answer 3 · 2021-12-27T20:11:09.390

Paramiko does not support recursive operations.

But it's easy to implement:

import os
from stat import S_ISDIR, S_ISREG

def get_r_portable(sftp, remotedir, localdir):
    for entry in sftp.listdir_attr(remotedir):
        remotepath = remotedir + "/" + entry.filename
        localpath = os.path.join(localdir, entry.filename)
        mode = entry.st_mode
        if S_ISDIR(mode):
            try:
                os.mkdir(localpath)
            except OSError:     
                pass
            get_r_portable(sftp, remotepath, localpath)
        elif S_ISREG(mode):
            sftp.get(remotepath, localpath)

You also can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See

Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.

But pysftp seems to be an abandoned project, so you better stick with Paramiko.

Krzysztof Maliszewski · Answer 4 · 2021-10-20T09:12:46.137

A small update to Dan LaManna's answer that works in 2021.

import paramiko
import os
from stat import S_ISDIR, S_ISREG    

def sftp_get_recursive(path, dest, sftp):
    item_list = sftp.listdir_attr(path)
    dest = str(dest)
    if not os.path.isdir(dest):
        os.makedirs(dest, exist_ok=True)
    for item in item_list:
        mode = item.st_mode
        if S_ISDIR(mode):
            sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
        else:
            sftp.get(path + "/" + item.filename, dest + "/" + item.filename)

transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp_get_recursive(remote_path, local_path, sftp)
sftp.close()

score 1 · Answer 5 · answered Jul 13 '11 at 12:05

1

You can use the stat() method of your sftp object:

http://www.lag.net/paramiko/docs/paramiko.SFTPClient-class.html

answered Jul 13 '11 at 12:05

guettli

25,042
81
346
663

Ah yes, that might be it. Haven't tested it yet but with a bit of luck the directory flag can be parsed out of SFTPAttributes.st_mode. Now I just need to research what flags are hidden in that mode value... – Arahman Jul 13 '11 at 12:51

user1268328 · Answer 6 · 2012-03-14T09:47:15.733

1

stat() method among other attributes returns permissions. d (for example drwxrwxrwx) shows that it is directory.

As example:

dir = oct(sftp.stat(path).st_mode)
print dir[0:2]

output interpritation: 01 fifo 02 character special 04 directory 06 block special 10 regular file 12 symbolic link 14 socket

edited Mar 14 '12 at 09:47

answered Mar 14 '12 at 07:44

user1268328

19
2

score 0 · Answer 7 · answered Dec 09 '22 at 10:41

Here is an answer for 2022.

pysftp is abandoned. paramiko does not implement recursivity for SFTP. So I made a library named sftputil (to make a parallel with ftputil) based on Paramiko.

sftputil implements functionalities such as walk, glob and sync. To copy an entire directory, the easiest way is to use the synchronization feature :

from sftputil import SFTP

sftp = SFTP("hostname", "username", password="password")

# To copy recursively
sftp.sync_pull("remote/path", "local/dir")

# To copy only the files
sftp.sync_pull("remote/path", "local/dir", recursive=False)

score -6 · Answer 8 · answered Jul 13 '11 at 08:15

-6

If u using Linux or Unix-like. U can use 'file' utility with popen. Or simple u can use os.path.isdir() =)

answered Jul 13 '11 at 08:15

Denis

7,127
8
37
58

4

Proper english please. Besides that, using `file` to test if something is a folder or not is pretty messy. And using `os.path.isdir()` on a remote file..? – ThiefMaster Jul 13 '11 at 12:10

Recursive directory download with Paramiko?

8 Answers8

Linked

Related