3

SO,

For some reason the little bit of code below is taking SIGNIFICANTLY longer to upload than if I was to upload via a FTP program or via the website itself.

The files come to 600kb which via ftp programs/directly via the web takes 1-3 seconds max however through ftplib it takes 15-30 seconds... that's over 15 times slower

The files that I'm uploading are single line html/php files (minified), is there a faster way to upload...?

In my actual code, I am uploading 25+ files all coming to a max of 1mb in total file size, is there a batch upload method with the .tmp renaming (prevents partially updated pages from loading) which would work better for me?

import ftplib

print "Uploading FULL.html"
## Login using the ftplib library and set the session as the variable ftp_session
ftp_session = ftplib.FTP('ftp.website.com','admin@website.com','Password1','',60)
## Open a file to upload
ftp_file = open('OUTPUT/DATA/FULL.html','r')
## Open a folder in the ftp server
ftp_session.cwd("/DATA/")
## Send/upload the file to the ftp server
ftp_session.storlines('STOR FULL.html.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("FULL.html.tmp", "FULL.html")

print "Uploading index.php"
## Open a file to upload
ftp_file = open('OUTPUT/index.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR index.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("index.php.tmp", "index.php")

print "Uploading secondary.php"
## Open a file to upload
ftp_file = open('OUTPUT/secondary.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR secondary.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("secondary.php.tmp", "secondary.php")

## Quit out of the FTP session
ftp_session.quit()

Example upload output....

Running...
Uploaded MAIN.html (284KB) in 9.677 seconds at 29.40 kbps
Uploaded GROUP1.html (12KB) in 0.660 seconds at 19.12 kbps
Uploaded GROUP10.html (9KB) in 1.137 seconds at 8.07 kbps
Uploaded GROUP11.html (14KB) in 1.014 seconds at 14.56 kbps
Uploaded GROUP12.html (17KB) in 0.913 seconds at 18.87 kbps
Uploaded GROUP13.html (12KB) in 0.675 seconds at 18.66 kbps
Uploaded GROUP14.html (13KB) in 0.738 seconds at 18.54 kbps
Uploaded GROUP15.html (16KB) in 0.971 seconds at 16.53 kbps
Uploaded GROUP16.html (14KB) in 1.755 seconds at 8.44 kbps
Uploaded GROUP17.html (13KB) in 1.504 seconds at 9.05 kbps
Uploaded GROUP18.html (14KB) in 0.725 seconds at 20.47 kbps
Uploaded GROUP19.html (14KB) in 1.691 seconds at 8.76 kbps
Uploaded GROUP2.html (10KB) in 0.541 seconds at 18.90 kbps
Uploaded GROUP20.html (13KB) in 0.799 seconds at 16.69 kbps
Uploaded GROUP21.html (14KB) in 0.713 seconds at 20.42 kbps
Uploaded GROUP22.html (14KB) in 0.776 seconds at 18.61 kbps
Uploaded GROUP23.html (14KB) in 0.592 seconds at 24.99 kbps
Uploaded GROUP24.html (13KB) in 0.734 seconds at 18.86 kbps
Uploaded GROUP25.html (20KB) in 0.845 seconds at 24.49 kbps
Uploaded GROUP26.html (18KB) in 0.637 seconds at 28.91 kbps
Uploaded GROUP27.html (11KB) in 0.781 seconds at 14.54 kbps
Uploaded GROUP28.html (16KB) in 0.798 seconds at 20.25 kbps
Uploaded GROUP29.html (12KB) in 0.542 seconds at 23.46 kbps
Uploaded GROUP3.html (16KB) in 0.585 seconds at 27.47 kbps
Uploaded GROUP4.html (14KB) in 0.622 seconds at 23.97 kbps
Uploaded GROUP5.html (14KB) in 0.716 seconds at 20.54 kbps
Uploaded GROUP6.html (18KB) in 1.109 seconds at 16.48 kbps
Uploaded GROUP7.html (11KB) in 0.654 seconds at 17.52 kbps
Uploaded GROUP8.html (9KB) in 0.560 seconds at 16.25 kbps
Uploaded GROUP9.html (12KB) in 0.582 seconds at 21.42 kbps
The whole routine took 72.468 seconds
Ryflex
  • 5,559
  • 25
  • 79
  • 148
  • Try using a [sniffer](http://en.wikipedia.org/wiki/Packet_analyzer) to dump the `ftplib` network traffic and examine it and/or compare to the manual case. – Vasiliy Faronov Dec 18 '13 at 22:15
  • @VasiliyFaronov It's definately but the using the sniffers I can't find anything wrong. I tried wireshark, caspa, snoop and a few others but nothing shows me what's wrong. Is there an easy way do something like zip the files and upload the zip and extract (idk any way to extract) – Ryflex Dec 22 '13 at 00:59
  • 1
    I would guess that the reason for your slowdown is the renaming. Did you test without that? – Chronial Dec 27 '13 at 05:29

2 Answers2

2

There are several reasons that your transfer is about as slow as possible and the application is sending much faster:

  1. You are sending one line at a time, the ftp application will send one block of 8192 bytes at a time,
  2. You are sending in text mode, the application is probably sending in binary compressed mode,

Unless there is a problem with line endings use open(filename, "rb") then tp_session.storbinary('STOR index.php.tmp', ftp_file) you can also try sending the command 'MODE COMPRESSED' and checking the reply and you could play with the block size to find the optimum.

The other thing I would set up would be something like:

import glob, os, ftplib
from time import time

sourcedir = '/where/your/files/are/locally'
destdir = '/where/to/put/them'
##    filelist = ['file2.ext', 'file2.ext', 'etc']
filelist = glob.glob(os.path.join(sourcedir, '*.php'))

ftp_session = ftplib.FTP('ftp.website.com','admin@website.com','password123','',60)
ftp_session.cwd(destdir)

for fname in filelist:
    filename = fname.replace(sourcedir+'\\', "")
    ftp_file = open(fname,'rb')
    result = ftp_session.storbinary('STOR %s.tmp' % filename, ftp_file)
    ftp_file.close()
    if 199 < int(result[0:3]) < 300: # Positive Completion
        ftp_session.rename('%s.tmp' % filename, filename)
    else:
        print "Upload of %s resulted in code %s" % (filename, result)
        break

You could potentially zip or tar.gz your files and send that but you would need a secure socket connection and appropriate permissions to unpack separately.

Ryflex
  • 5,559
  • 25
  • 79
  • 148
Steve Barnes
  • 27,618
  • 6
  • 63
  • 73
  • On running your code snippit, I got the following error: `if 199 < int(result) < 300: # Positive Completion ValueError: invalid literal for int() with base 10: '226-File successfully transferred\n226 13.494 seconds (measured here), 21.09 Kbytes per second'` – Ryflex Jan 02 '14 at 20:39
  • As for compression, I can't find anywhere in the docs about compression and as for sending in storbinary I was having issues with it cutting off a bit of some html/php files. – Ryflex Jan 02 '14 at 20:45
  • Minor change to isolate the number above may help with the error - for some details on compression see https://en.wikipedia.org/wiki/Mode_Z_compression – Steve Barnes Jan 02 '14 at 21:46
  • Thanks for the minor edit, I've edited your post for review chaning a few things like imports, a mistake with a string format and a few other changes. – Ryflex Jan 02 '14 at 21:50
  • The closest thing I found for the compression is: http://stackoverflow.com/questions/1184844/downloading-text-files-with-python-and-ftplib-ftp-from-z-os however it's pretty confusing and I'm unsure how to impliment it. – Ryflex Jan 02 '14 at 21:53
  • 1
    Uploaded 30 files over `storlines` and it took 60-70seconds each, where as with `storbinary` it took 63-69 seconds, over all of the times their average upload time was 66seconds and that's for a total of ~500kb. – Ryflex Jan 02 '14 at 21:55
0

define blocksize along with storbinary of ftp connection,so you will get 1.5-3.0x more faster connection than FTP Filezilla :)

from ftplib import FTP

USER = "Your_user_id"
PASS = "Your_password"
PORT = 21
SERVER = 'ftp.billionuploads.com' #use FTP server name here

ftp = FTP()
ftp.connect(SERVER, PORT)
ftp.login(USER, PASS)

try:
    file = open(r'C:\Python27\1.jpg','rb')
    ftp.storbinary('STOR ' + '1.jpg', file,102400) #here we store file in 100kb blocksize
    ftp.quit()
    file.close()
    print "File transfered"
except:
    print "Error in File transfering"
sohom
  • 577
  • 4
  • 7