5

I was wondering whether there was a best practice for checking if an upload to your ftp server was successful.

The system I'm working with has an upload directory which contains subdirectories for every user where the files are uploaded.

Files in these directories are only temporary, they're disposed of once handled.

The system loops through each of these subdirectories and new files in them and for each file checks whether it's been modified for 10 seconds. If it hasn't been modified for 10 seconds the system assumed the file was uploaded successfully.

I don't like the way the system currently handles these situations, because it will try and handle the file and fail if the file upload was incomplete, instead of waiting and allowing the user to resume the upload until it's complete. It might be fine for small files which doesn't take a lot of time to upload, but if the file is big I'd like to be able to resume the upload.

I also don't like the loops of directories and files, the system idles at a high cpu usage, so I've implemented pyinotify to trigger an action when a file is written. I haven't really looked at the source code, I can only assume it is more optimized than the current implementation (which does more than I've described).

However I still need to check whether the file was successfully uploaded.

I know I can parse the xferlog to get all complete uploads. Like: 

awk '($12 ~ /^i$/ && $NF ~ /^c$/){print $9}' /var/log/proftpd/xferlog

This would make pyinotify unnecessary since I can get the path for complete and incomplete uploads if I only tail the log.

So my solution would be to check the xferlog in my run-loop and only handle complete files.

Unless there's a best practice or simply a better way to do this?

What would the disadvantages be with this method?

I run my app on a debian server and proftpd is installed on the same server. Also, I have no control over clients sending the file.

Mike Causer
  • 8,196
  • 2
  • 43
  • 63
JayDL
  • 215
  • 5
  • 12
  • What is the essential purpose of the python script? Is it cleaning out used-up files, show status on a webpage, or something different? – Alfred Bratterud Sep 10 '12 at 05:31
  • Reading this now, a year after posted, the question does seem a but unclear. Basically I wanted to do something with uploaded files once an upload is complete. But couldn't find a way to use proftpd to assure me of a complete upload without blocking FTP REST commands. So at the time of the question I ended up doing a solution with iNotify and validating the files when written. – JayDL Jan 03 '13 at 14:01

2 Answers2

8

Looking at the proftpd docs, I see http://www.proftpd.org/docs/directives/linked/config_ref_HiddenStores.html

The HiddenStores directive enables two-step file uploads: files are uploaded as ".in.filename." and once the upload is complete, renamed to just "filename". This provides a degree of atomicity and helps prevent 1) incomplete uploads and 2) files being used while they're still in the progress of being uploaded.

This should be the "better way" to solve the problem when you have control of proftpd as it handles all the work for you - you can assume that any file which doesn't start .in. is a completed upload. You can also safely delete any orphan .in.* files after some arbitrary period of inactivity in a tidy-up script somewhere.

moopet
  • 6,014
  • 1
  • 29
  • 36
  • I saw that, but unfortunately I can't block REST commands. At the time when I asked this question, I ended up doing a solution with pyinotify and validating the written file before handling it. But would be great if proftpd could handle this for me regardless if REST or APPE commands are being used. – JayDL Jan 03 '13 at 13:54
  • We could probably make ProFTPD support `HiddenStores` functionality when `REST` or `APPE` are used. However, there are some possible consequences. When `REST` or `APPE` are used, there's an existing file; that existing file would need to _copied_ as the temporary file. Then, when the `STOR` (after the `REST` command) or `APPE` completes, that file is renamed into place, *replacing* the existing file. If _any other client_ was similarly modifying that same file at the same time, there's a possible race -- who renames the file last wins. Would that be acceptable behavior for your needs? – Castaglia Jan 10 '16 at 18:59
0

You can use pure-uploadscript if your pure-ftpd installation was compiled with --with-uploadscript option.

It is used to launch a specified script after every upload is completely finished.

  1. Set CallUploadScript to "yes"
  2. Make a script with a command like touch /tmp/script.sh

  3. Write the code in it. In my example the script renames the file and adds ".completed" before the file name:

    #!/bin/bash fullpath=$1 filename=$(basename "$1") dirname=${fullpath%/*} mv "$fullpath" "$dirname/completed.$filename"

  4. Run chmod 755 /tmp/script.shto make the script executable by pure-uploadscript

  5. Then run a command pure-uploadscript -B -r /etc/pure-ftpd/uploadscript.sh

Now /tmp/script.sh will be launched after each completed upload.

Jeffrey Bosboom
  • 13,313
  • 16
  • 79
  • 92
SlashMan
  • 19
  • 2