Avoid Subprocess Timeouts

Question

Okay, so I'm currently using a python script to upload multiple files to a given server at once.

I'm running python version 2.7.3.

def upload(t):
    server="server"
    user="user"
    password="password"
    service="service"
    ftp=ftplib.FTP(server)
    ftp.login(user=user,passwd=password,acct="")
    ftp.storbinary("STOR "+t.split('/')[-1], open(t,"rb"))
    print "{} has been uploaded to {}".format(t.split("/")[-1],service)
def ftp_upload(t=files):
    server="server"
    user="user"
    password="password"
    service="service"
    ftp.login(user=user,passwd=password,acct="")
    pool=multiprocessing.Pool(processes=4)
    pool.map(upload,t)

However, I am getting error messages as timeouts with sockets occur or something (which doesn't happen when I just use the ftp console command).

It seems like subprocess does something different? I don't even particularly care about monitoring it; in fact, I'd rather the process not terminate if my connection to the server is messed with. Any reporting is gravy.

Error Message:

 File "/usr/bin/jat", line 301, in ftp_upload
    pool.map(upload,files_to_upload)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
socket.timeout: timed out

Please post the precise error message instead of `or something`. Thanks! — phihag, Jan 10 '13 at 17:46

score 0 · Answer 1 · edited May 23 '17 at 11:57

pool.Map() propagates exceptions from the subprocesses it manages to the main process. Notice that the exceptions are being reported from multiprocessing/pool.py, and the traceback ends with "raise self._value".

That line indicates that the socket.timeout is being raised in one of the subprocesses. I suspect it's your ftp.login that is timing out. According to the ftplib documentation, the global default timeout is used by an FTP object if you don't specify one when you create the object.

On a side note, I would suggest using one of the approaches suggested by the answerers of this question to pass a bit more detail back to your main thread when an exception occurs in a subprocess.

Avoid Subprocess Timeouts

1 Answers1