ftplib - file creation very slow: SSLError: The read operation timed out

Question

import ftplib
import csv
import StringIO

sio = StringIO.StringIO()
data = csv.writer(sio)
data.writerows(data_to_export)
sio.seek(0)
ftps.storbinary("STOR " + 'blah.csv', sio)
ftps.close()

So the file I'm creating is only 56 lines, it creates very slowly. And when it finishes creating, I get this error:

    ftps.storbinary("STOR " + 'blah.csv', sio)
  File "/usr/lib/python2.7/ftplib.py", line 752, in storbinary
    conn.unwrap()
  File "/usr/lib/python2.7/ssl.py", line 384, in unwrap
    s = self._sslobj.shutdown()
SSLError: The read operation timed out

I don't get it why it is so slow and why it gives time out, even though it creates that file.

P.S. please ask if I need to provide additional information

Update I also tried cString, but it does not improve anything as I guess its not related with writing speed.

If there is any difference, connection to ftp is made using Implicit SSL/TLS mode (that ftp does not support Explicit SSL/TLS).

Update Digging deeper. That is what debugging shows for ftp:

*cmd* 'TYPE I'
*resp* '200 Binary mode selected.'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (*,*,*,*,*,*)'
*cmd* 'STOR blah.csv'
*resp* '125 Secure data connection open; transfer starting.'

And then stucks at that last output till I get connection time out.

Note. My internet connection is very good, but that ftp is kind of slow. Still I guess such small file should be handled much faster than this.

Update2 That is some weird ftp. Trying unsecure ftp connection, files are uploaded properly and fast. That Implicit connection has something to do with this slow performance and time outs.

Answer 1

In regards to speed there are too many factors to guess at, so i would adjust the code and try the following code which i recently created for customer ftp delivery and can state it works without error.

Note this is from a much larger file but have extracted the relevant sections so ignore some of the variables ie args.var, you may have to adjust the blocksize dependant on bandwidth etc try taking it down to 1024 then working up.

Hope it helps

def initiate_ftp_connection(ftp_host, user, passwd, ftp_dir):
    ftp_session = ftplib.FTP()
    ftp_session.connect(ftp_host, 21)
    #uncomment for debugging.
    #ftp_session.set_debuglevel(2)
    ftp_session.login(user=user, 
                      passwd=passwd)
    #cd to correct remote directory
    ftp_session.cwd(ftp_dir)
    return ftp_session



def upload_deliverables(session, file_and_path):
    working_dir = os.path.dirname(file_and_path)
    #strip to remove any newlines
    filename = os.path.basename(file_and_path).strip()
    totalSize = os.path.getsize(file_and_path)
    #instantiate progress tracker for status updates
    uploadTracker = FtpUploadTracker(int(totalSize),filename)
    #change dir to working_dir
    os.chdir(working_dir)
    '''
     Trigger the ftp upload (storbinary) for the deliverable.
      Args:
       1: FTP KEYWORD and FILE
       2: File IO
       3: Blocksize
       4: Callback
    '''
    session.storbinary('STOR ' + filename, 
                        open(filename,'r'), 
                        8192, 
                        uploadTracker.ftp_callback)


#connect to server
ftp_session = initiate_ftp_connection(args.ftp_host, 
                                      args.ftp_user, 
                                      args.ftp_pass,
                                      args.ftp_dir)

#start ftp delivery
upload_deliverables(ftp_session, args.asset)
#quit the ftp session
ftp_session.quit()   
#close any file handles.
ftp_session.close()

Answer 2

I faced this issue on STORBINARY function when using python's ftplib.FTP_TLS, prot_p and Microsoft FTP server.

Example:

ftps = FTP_TLS(host,username,password)
ftps.prot_p
STORBINARY...

The error indicated timeout on the unwrap function.

It is related to the following issues:

https://www.sami-lehtinen.net/blog/python-32-ms-ftps-ssl-tls-lockup-fix

https://bugs.python.org/issue10808

https://bugs.python.org/issue34557

Resolution:

1. Open the python page for ftplib: https://docs.python.org/3/library/ftplib.html

2. Click on the source code which will take you to something like this: https://github.com/python/cpython/blob/3.10/Lib/ftplib.py

3. Create a copy of this code into your project (example: my_lib\my_ftplib.py)

4. For the method that is failing, in your case STORBINARY, the error looks to be on the line where it says conn.unwrap() in that method. Comment this line. Enter keyword pass otherwise the empty if block will give syntax error.

5. Import the above library in your file where you are instantiating the FTP_TLS. Now you will no longer face this error.

Reasoning: The code in the function def ntransfercmd (under FTP_LTS class) encloses the conn object into a SSL session. The above line which you have commented is responsible for tearing down the SSL session post transfer. For some reason, when using Microsoft's FTP server, the code gets blocked on that line and results in timeout. This can be either because post transfer the server drops the connection or maybe the server unwraps the SSL from its side. I am not sure. Commenting that line is harmless because eventually the connection will be closed anyways - see below for details:

In ftplib's python code, you will notice that the conn object in STORBINARY function is enclosed in a with block, and that it is created using socket.create_connection. This means that .close() is automatically called when the code exits the with block (you can confirm this by looking at the __exit__ method on source code of python's socket class).