1

I am downloading a ~300 Mb file through an ftp server periodically every 6 hours or so. Most downloads go well, but sometimes the process hangs and I need to kill and restart manually. So I want a more robust download system, preferably with the following criteria.

  1. Avoids timeouts or hangs as much as possible. And can deal with them if they happen
  2. If the download is killed, try resuming it a few times until completed (or send error message if it didn't work for any of the times tried)

For (1), I read in this question that it would be good to use python threading with keep_alive calls until all blocks have been downloaded.

def downloadFile(…):
    ftp = FTP(…)
    sock = ftp.transfercmd('RETR ' + filename)
    def background():
        f = open(…)
        while True:
            block = sock.recv(1024*1024)
            if not block:
                break
            f.write(block)
        sock.close()
    t = threading.Thread(target=background)
    t.start()
    while t.is_alive():
        t.join(60)
        ftp.voidcmd('NOOP')

For (2), there could be a loop that checks if the file has been completely downloaded. And if not, it could restart from the point it left it as. Based on this question.

for i in range(3):
    if "Check if file has been completely downloaded":
        if os.path.exists(filename):
            restarg = {'rest': str(os.path.getsize(filename))}
        else:
            restarg = {}
        ftp.transfercmd("RETR " + filename, **restarg)

But how to combine (1) and (2)? Can you resume a threaded download? With many blocks which we don't even know in which order were downloaded..

If these two methods cannot be combined, do you have any other idea?

Also, I am not very sure how to tell if the ftp download was completed. Should I check the file size for this? File sizes might change from one download to another.

marcos
  • 93
  • 8
  • Are you sure this is [tag:pyth] question and not [tag:python]? + Did you positively test that `NOOP` helps in your scenario? – Martin Prikryl Sep 14 '20 at 06:12
  • Yes sorry, python! That is another issue, how can I test the code for a hang? Is there a way to break the connection for some time manually and then re-start? – marcos Sep 14 '20 at 06:39
  • OK, so I take it that actually the `NOOP` is useless for you, as you actually suffer a "hanging" issue, not disconnect issue. – Martin Prikryl Sep 14 '20 at 06:54
  • Where does the code hang? – Martin Prikryl Sep 14 '20 at 06:59
  • It usually hangs at the end of the transfer, I can see that the file is already the size I expected – marcos Sep 14 '20 at 07:40
  • Where *in the code*? – Martin Prikryl Sep 14 '20 at 07:53
  • It hanged when I used `ftp.retrbinary("RETR " + downloadedFileName, open(downloadedFilePath, 'wb').write)`. That is why I'm looking for a new method, which is the question I am asking – marcos Sep 14 '20 at 18:59
  • And does it hang, when you use your `downloadFile`? – Martin Prikryl Sep 14 '20 at 19:18
  • So far no, it works well. But I don't really know how to test for a hang. Do I just need to let it run 100 times and see if it works? If you have any advice on how to test it or in the code above I would be happy to hear it! – marcos Sep 15 '20 at 14:36

0 Answers0