How to determing if a file has finished downloading in python

Question

I have a folder called raw_files. Very large files (~100GB files) from several sources will be uploaded to this folder.

I need to get file information from videos that have finished uploading to the folder. What is the best way to determine if a file is currently being downloaded to the folder (pass) or if the video has finished download (run script)? Thank you.

Your upload software should notify (or run) your script. There's no platform-independent way of telling whether a file is "finished". — Fred Foo, Sep 21 '11 at 12:52
I think your question has been answered in [this stackoverflow post](http://stackoverflow.com/a/18744072/3511819). — Alex, Jun 08 '16 at 03:39

score 2 · Answer 1 · answered Sep 21 '11 at 13:02

2

The most reliable way is to modify the uploading software if you can.

A typical scheme would be to first upload each file into a temporary directory on the same filesystem, and move to the final location when the upload is finished. Such a "move" operation is cheap and atomic.

A variation on this theme is to upload each file under a temporary name (e.g. file.dat.incomplete instead of file.dat) and then rename. You script will simply need to skip files called *.incomplete.

answered Sep 21 '11 at 13:02

NPE

486,780
108
951
1,012

Note that moving a file is not atomic on Windows. – Sven Marnach Sep 21 '11 at 13:07
@Sven Marnach: Thanks for the comment. I'd be interested to hear about an actual real-life scenario when a file move **within the same filesystem** weren't in effect atomic on Windows. Besides, there's `MoveFileTransacted`. – NPE Sep 21 '11 at 13:13
See http://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows. I don't think this could introduce any race conditions in the case at hand, though. – Sven Marnach Sep 21 '11 at 13:17
@aix: thank you for the response. One limitation I have here is that I am not the one uploading the files. The files are coming from multiple sources. The only thing I have control over is the folder they upload to. How would this change the scenario? – David542 Sep 23 '11 at 08:44

score 0 · Accepted Answer · answered Sep 21 '11 at 12:56

If you check those files, store the size of the files somewhere. When you are in the next round and the filesize is still the same, you can pretty much consider them as finished (depending on how much time is between first and second check). The time interval could e.g. be set to the timeout-interval of your uploading service(FTP, whatever).

There is no special sign or content showing that a file is complete.

How to determing if a file has finished downloading in python

2 Answers2