2

I'm trying to upload an image to my uploads folder on my remote server. The folder structure is always uploads/year/month/ and I can't get paramiko to check if the folders exist and if not make them.

SSH connection is working, uploading a file is working too, but creating the subfolders in the uploads directory isn't working.

I came across what looked like the solution here. It's the same question I have, but I'm on iOS and use Pythonista. Option A: my code is plain wrong or Option B it's an iOS/Pythonista specific issue.

So, the code from the other thread (linked above) set a definition and runs a try/error loop to tests if the folders passed through it already exists and if not creates them. In my script below it's # Set Definition for "mkdir -p".

Calling it with the remoteFilePath…

  1. Unnecessary because: ideally it should only test if datePath exists, since the remotePath definitely exists
  2. Likely problematic because: fileName is no path and will be put there by the next command.

I tried adjusting the script, but somehow I can't make it work.

I get errors no matter what I try:

  • with version 1: TypeError: mkdir_p() takes exactly 2 arguments (1 given)"
  • with version 2: AttributeError: 'tulpe' object has no attribute 'rfind'
  • with version 3: Exception: unknown type for (/home/userZ/Dropbox/uploads/year/month', 'test.png') type <type 'tuple'>

Here's a snippet of the relevant parts of the script (or a gist if you prefer the look of it):

# Set Variables
fileName = "temp.png"
remotePath = "/home/userZ/Dropbox/uploads/"
datePath = "year/month/"
remoteFilePath =  remotePath + datePath + fileName #

# Set Definition for "mkdir -p"
def mkdir_p(sftp,remote_directory):
    remote_dirname, basename = os.path.split(remote_directory)
    mkdir_p(os.path.dirname(remote_directory))
    try:
        sftp.chdir(name)
    except IOError:
        sftp.mkdir(name)
        sftp.chdir(name)

try:
    transport.connect(username = username, password = password)
    sftp = paramiko.SFTPClient.from_transport(transport)     # Start SFTP client

    # Try to make remote path - 3 Versions and all fail
    mkdir_p(sftp,remoteFilePath) # Version 1
    #mkdir_p(sftp, os.path.split(remoteFilePath)) # Version 2
    #sftp.mkdir(os.path.split(remoteFilePath)) # Version 3

    # Put file to remote
    sftp.put('temp.png', remoteFilePath)

    # Close connection
    finally:
        transport.close()
        sftp.close()

Any help is appreciated. (Careful: OP = Python noob). I rely on Paramiko because my shared host only supports SFTP. Otherwise I'd have gone with FTPlib.

Community
  • 1
  • 1
patrick
  • 574
  • 3
  • 10
  • 25

5 Answers5

2

You run into the same problem with FTP, you can't mkdir /some/long/path, you have to

cd /
mkdir some # ignore errors if it exists
cd some
mkdir long # ignore errors if it exists
cd  long
mkdir path # ignore errors if it exists
cd path

I'm guessing that the historical reason for this model is to provide a simple boolean SUCCEED/FAIL for any command issued by the client.

Having some fuzzy error like 'Couldn't make directory /some/long/path because directory /some/long doesn't exist' would be a pain in the ass to handle on the client side.

In looking at the SFTP protocol spec re: filenames ( https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#page-15 ), it's unclear to me if clients are expected to understand '/' as a path separator in the context of making a directory - since you could have an SFTP server on a windows box as well, paramiko may just omit support for this entirely?

tl;dr: Modify your code to split the path up into it's individual components, try to chdir to them, if that fails, try to make them, if that fails you're dead, otherwise chdir to the next subdirectory and continue until you have no more subdirectories to create.

EDIT

The specific problem you're having with the line

mkdir_p(os.path.split(remoteFilePath))

can be fixed by changing it to

mkdir_p(sftp, os.path.split(remoteFilePath))
Community
  • 1
  • 1
synthesizerpatel
  • 27,321
  • 5
  • 74
  • 91
  • Mh. To be frank, I *thought* I already did that by adding the definition from the other thread. I edited my question to make it clearer where I think I went wrong. – patrick Aug 05 '13 at 07:01
  • Updated my answer to resolve your issue. Simple syntax error. – synthesizerpatel Aug 05 '13 at 08:55
  • I tried it this way and get an error (see edit above - Version 2): AttributeError: 'tulpe' object has no attribute 'rfind' /// Thanks for the help, I keep trying today, just wrapped a wet towel around my head to stop it from steaming too much. – patrick Aug 05 '13 at 09:28
  • I think I understand the problem - the split is causing the issue. Instead of **mkdir_p(sftp, os.path.split(remote_directory))**, try **mkdir_p(sftp, remote_dirname)**. If that doesn't fix it, review the original mkdir_p code and make sure you've got it cut/pasted right. – synthesizerpatel Aug 05 '13 at 14:11
1

CClauss got the answer and put it in the comment section of the gist linked above. I take no credit.

This was his answer - in cause anyone else tries to emulate mkdir -p with paramiko, then here you go:

My sense is that you need to: Try to mkdir on :/home/ Try to mkdir on :/home/userZ/ Try to mkdir on :/home/userZ/Dropbox/ Try to mkdir on :/home/userZ/Dropbox/uploads/ Try to mkdir on :/home/userZ/Dropbox/uploads/year/ Try to mkdir on :/home/userZ/Dropbox/uploads/year/month/ Then cd to /home/userZ/Dropbox/uploads/year/month/ Then copy your file

Try this...

# Slash '/' is hardcoded because ftp always uses slash
def mk_each_dir(sftp, inRemoteDir):
    currentDir = '/'
    for dirElement in inRemoteDir.split('/'):
        if dirElement:
            currentDir += dirElement + '/'
            print('Try to mkdir on :' + currentDir)
            try:
                sftp.mkdir(currentDir)
            except:
                pass # fail silently if remote directory already exists

# Set Variables
fileName = "temp.png"
remotePath = "/home/userZ/Dropbox/uploads/"
datePath = "year/month/"
remoteDirPath =  remotePath + datePath
mk_each_dir(sftp, remoteDirPath)
sftp.chdir(remoteDirPath)
remoteFilePath =  remoteDirPath + fileName
patrick
  • 574
  • 3
  • 10
  • 25
1
import os

def mkpath(path, sftp):
    """mkdir -p helper for paramiko

    Args:
        path: string of path to create
        sftp: paramiko sftp client
    """
    path = os.path.dirname(path)
    if len(path) > 1:
        try:
            sftp.mkdir(path)
        except IOError:
            mkpath(path, sftp)

Something like this solved the problem* for me. I call mkpath before doing an sftp.put on the full remotepath argument.

*Where my problem was sftp.mkdir can't make directories recursively.

Wyrmwood
  • 3,340
  • 29
  • 33
0

Yes something like the below does the trick for you. This is what I have been using for a couple of years now. Like Cclaus said, you need to create each path before trying to create its sub directories. createPath() just does the magic you need. 

def ftpWrite(sftpConn, localpath, remotepath):
    createPath(sftpConn, remotepath)
    sftpConn.put(localpath, remotepath)

def createPath(sftpConn, remotepath):
    parts = remotepath.split('/')
    for n in range(3, len(parts)):
        path = '/'.join(parts[:n])       
        try:
            sftpConn.stat(path)
        except:
            sftpConn.mkdir(path)
Michael Dunlap
  • 4,300
  • 25
  • 36
ggog
  • 13
  • 4
0

Function for creating subfolders chain if some of folders doesn't exist (via Paramiko SFTP Connection object)

def folder_creator(path):

    paramiko_ssh = paramiko.SSHClient()
    # additional paramiko manipulations if required here
    paramiko_sftp = paramiko_ssh.open_sftp()

    path_parts = path.split('/')
    first_dir = '/'.join(path_parts[0:2])
    all_parts = [first_dir] + path_parts[2:]

    for dir in all_parts:
            try:
                    paramiko_sftp.chdir(dir)
            except IOError:
                    paramiko_sftp.mkdir(dir)
                    paramiko_sftp.chdir(dir)

    paramiko_ssh.close()
AlGiorgio
  • 497
  • 5
  • 25