Upload files using SFTP in Python, but create directories if path doesn't exist

Question

I want to upload a file on a remote server with Python. I'd like to check beforehand if the remote path is really existing, and if it isn't, to create it. In pseudocode:

if(remote_path not exist):
    create_path(remote_path)
upload_file(local_file, remote_path)

I was thinking about executing a command in Paramiko to create the path (e.g. mkdir -p remote_path). I came up with this:

# I didn't test this code

import paramiko, sys

ssh = paramiko.SSHClient()
ssh.connect(myhost, 22, myusername, mypassword)
ssh.exec_command('mkdir -p ' + remote_path)
ssh.close

transport = paramiko.Transport((myhost, 22))
transport.connect(username = myusername, password = mypassword)

sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(local_path, remote_path)
sftp.close()

transport.close()

But this solution doesn't sound good to me, because I close the connection and then reopen it again. Is there a better way to do it?

related: [os.renames for ftp in python](http://stackoverflow.com/q/14641267/4279) — jfs, Feb 15 '13 at 18:20

score 63 · Accepted Answer · edited Feb 25 '15 at 21:20

63

SFTP supports the usual FTP commands (chdir, mkdir, etc...), so use those:

sftp = paramiko.SFTPClient.from_transport(transport)
try:
    sftp.chdir(remote_path)  # Test if remote_path exists
except IOError:
    sftp.mkdir(remote_path)  # Create remote_path
    sftp.chdir(remote_path)
sftp.put(local_path, '.')    # At this point, you are in remote_path in either case
sftp.close()

To fully emulate mkdir -p, you can work through remote_path recursively:

import os.path

def mkdir_p(sftp, remote_directory):
    """Change to this directory, recursively making new folders if needed.
    Returns True if any folders were created."""
    if remote_directory == '/':
        # absolute path so change directory to root
        sftp.chdir('/')
        return
    if remote_directory == '':
        # top-level relative directory must exist
        return
    try:
        sftp.chdir(remote_directory) # sub-directory exists
    except IOError:
        dirname, basename = os.path.split(remote_directory.rstrip('/'))
        mkdir_p(sftp, dirname) # make parent directories
        sftp.mkdir(basename) # sub-directory missing, so created it
        sftp.chdir(basename)
        return True

sftp = paramiko.SFTPClient.from_transport(transport)
mkdir_p(sftp, remote_path) 
sftp.put(local_path, '.')    # At this point, you are in remote_path
sftp.close()

Of course, if remote_path also contains a remote file name, then it needs to be split off, the directory being passed to mkdir_p and the filename used instead of '.' in sftp.put.

edited Feb 25 '15 at 21:20

Oleh Prypin

33,184
10
89
99

answered Feb 11 '13 at 19:57

isedev

18,848
3
60
59

it doesn't handle non-existing parent directories (`-p`). Compare os.mkdir() vs. os.makedirs(). Split the path and make the recursive call to create parent directories if necessary – jfs Feb 11 '13 at 20:09
in the function mkdir_p there's no handle to sftp – franzlorenzon Feb 12 '13 at 09:16
I found another problem. When uploading the file, it starts from the home directory. For example, if I want to put a file in /var/www/temp/, it upload it to /home/user/var/www/temp/ . With this correction it works for me: `if remote_directory == '/' or remote_directory == '': if remote_directory == '/': sftp_client.chdir('/') ` . Moreover, I found that using os.path.split is more pythonic, maybe :) `remote_dirname, basename = os.path.split(remote_directory) mkdir_p(sftp_client, remote_dirname)` – franzlorenzon Feb 12 '13 at 09:30
1

good point. updated accordingly (not 100% sure about the more Pythonic statement though ;P) – isedev Feb 13 '13 at 01:17
Yes, maybe it's a matter of taste :) – franzlorenzon Feb 13 '13 at 08:02
2

you should use `posixpath` instead of `os.path` for ftp paths. You could avoid visiting all path segments by [moving the recursive call into the exception handler](http://stackoverflow.com/a/14645743/4279) – jfs Feb 15 '13 at 18:19
@J.F.Sebastian -- posixpath? enlighten me! – isedev Feb 15 '13 at 18:36

gabhijit · Answer 2 · 2016-01-15T04:51:29.127

12

Something simpler and slightly more readable too

def mkdir_p(sftp, remote, is_dir=False):
    """
    emulates mkdir_p if required. 
    sftp - is a valid sftp object
    remote - remote path to create. 
    """
    dirs_ = []
    if is_dir:
        dir_ = remote
    else:
        dir_, basename = os.path.split(remote)
    while len(dir_) > 1:
        dirs_.append(dir_)
        dir_, _  = os.path.split(dir_)

    if len(dir_) == 1 and not dir_.startswith("/"): 
        dirs_.append(dir_) # For a remote path like y/x.txt 

    while len(dirs_):
        dir_ = dirs_.pop()
        try:
            sftp.stat(dir_)
        except:
            print "making ... dir",  dir_
            sftp.mkdir(dir_)

edited Jan 15 '16 at 04:51

answered Dec 06 '13 at 11:25

gabhijit

3,345
2
23
36

1

+1 for providing a non-recursive alternative. Notice that the "remote" input parameter here is a remote file path. If you want this function to have a remote directory path as input instead, replace "dir_, basename = os.path.split(remote)" with "dir_ = remote" . – Alan Evangelista Jan 14 '16 at 10:06
@AlanEvangelista Thanks for the comment. Updated the code that passes a flag `is_dir`. Please review and edit if required. – gabhijit Jan 15 '16 at 04:52
1

You shouldn't use `except:` to check for errors. See: https://stackoverflow.com/a/18982771/1113207 – Mikhail Gerasimov May 14 '20 at 08:08

score 7 · Answer 3 · answered Jan 28 '15 at 18:47

Had to do this today. Here is how I did it.

def mkdir_p(sftp, remote_directory):
    dir_path = str()
    for dir_folder in remote_directory.split("/"):
        if dir_folder == "":
            continue
        dir_path += r"/{0}".format(dir_folder)
        try:
            sftp.listdir(dir_path)
        except IOError:
            sftp.mkdir(dir_path)

score 5 · Answer 4 · answered Sep 24 '19 at 10:31

you can use pysftp package:

import pysftp as sftp

#used to pypass key login
cnopts = sftp.CnOpts()
cnopts.hostkeys = None

srv = sftp.Connection(host="10.2.2.2",username="ritesh",password="ritesh",cnopts=cnopts)
srv.makedirs("a3/a2/a1", mode=777)  # will happily make all non-existing directories

you can check this link for more details: https://pysftp.readthedocs.io/en/release_0.2.9/cookbook.html#pysftp-connection-mkdir

Mikhail Gerasimov · Answer 5 · 2020-05-14T11:29:48.030

2

My version:

def is_sftp_dir_exists(sftp, path):
    try:
        sftp.stat(path)
        return True
    except Exception:
        return False


def create_sftp_dir(sftp, path):
    try:
        sftp.mkdir(path)
    except IOError as exc:
        if not is_sftp_dir_exists(sftp, path):
            raise exc


def create_sftp_dir_recursive(sftp, path):
    parts = deque(Path(path).parts)

    to_create = Path()
    while parts:
        to_create /= parts.popleft()
        create_sftp_dir(sftp, str(to_create))

We try mkdir without trying listdir/stat first due to EAFP principle (it's also more performant to make one network request than several).

edited May 14 '20 at 11:29

answered May 14 '20 at 08:31

Mikhail Gerasimov

36,989
16
116
159

1

But this way, `create_sftp_dir` will seemingly succeed, if the directory does not exist and cannot be created. – Martin Prikryl May 14 '20 at 08:50
@MartinPrikryl you're correct, thanks. I fixed the code, it still tries to create a dir without pre-check and does the check only in case dir wasn't created to determine the reason. – Mikhail Gerasimov May 14 '20 at 11:31
OK, but now, if your are calling this with `/foo/bar` and both exist, your code will do four requests, comparing to one, if you first tested `/foo/bar` existence. – Martin Prikryl May 14 '20 at 11:35
Btw, are you sure the `Path` class will handle posix-style SFTP paths correctly, when used on Windows? – Martin Prikryl May 14 '20 at 11:35
"will do four requests, comparing to one" - to be precise - 4 to 2 (one to check + one to create). That's true only for the case when we try to create a dirs that already exist. For cases when many dirs don't exist, we will get more benefit more dirs don't exist. Number of requests also depends on how you check things: left-to-right or right-to-left, which is EAFP-indifferent. Good point about SFTP paths on Windows, thanks, I'll think about it further! – Mikhail Gerasimov May 14 '20 at 12:04

score 1 · Answer 6 · edited Dec 04 '14 at 19:03

1

Paramiko contains a mkdir function:

http://paramiko-docs.readthedocs.org/en/latest/api/sftp.html#paramiko.sftp_si.SFTPServerInterface.mkdir

edited Dec 04 '14 at 19:03

Felippe Raposo

431
4
23

answered Feb 11 '13 at 19:55

Benjamin

609
3
8

1

I know. But does this handle the case of a non existing path? – franzlorenzon Feb 11 '13 at 19:56

score 0 · Answer 7 · answered Dec 20 '18 at 23:59

Assuming sftp operations are expensive, I would go with:

def sftp_mkdir_p(sftp, remote_directory):
    dirs_exist = remote_directory.split('/')
    dirs_make = []
    # find level where dir doesn't exist
    while len(dirs_exist) > 0:
        try:
            sftp.listdir('/'.join(dirs_exist))
            break
        except IOError:
            value = dirs_exist.pop()
            if value == '':
                continue
            dirs_make.append(value)
        else:
            return False
    # ...and create dirs starting from that level
    for mdir in dirs_make[::-1]:
        dirs_exist.append(mdir)
        sftp.mkdir('/'.join(dirs_exist))```

Upload files using SFTP in Python, but create directories if path doesn't exist

7 Answers7

Linked