2

I'm writing a web application in python/pyramid that handles the upload of a file (via jquery uploader). The code for the upload is:

@view_config(route_name='file.upload', renderer='json')
def file_upload(request):

for item, filestorage in request.POST.items():
    f = File.create(filestorage)

    u = Upload.create(f.hash)

return {
    'url': request.route_url('file.get', uploadid=u.urlid)
}

where then File.create makes a sha1 of the file and moves it in a permanent location while putting the metadata in a database. (File is a SQLAlchemy class actually)

The problem here is that the view callable is called after that the file transfer to the server is complete. This poses two problems: the first one is that i'm unable to reject the file transfer if it's bigger than some size. The second is that i have to wait to receive the whole file and then i can start hashing it.

What i would like to obtain is to start process the file - some sort of stream - as soon i'm getting the data so i can hash it while the user is uploading it and i can stop the transfer if the size is bigger than some value.

Riccardo Cagnasso
  • 450
  • 5
  • 19

2 Answers2

3

In your case prefer client-side validation for file size limits. Using a jQuery plugin you require your clients to have Javascript enabled.

Look at pyramid application PyGall. It switched from using gp.fileupload to jQuery-File-Upload and injects file size limits from pyramid settings into jQuery plugin initialization. It computes md5 checksum as well and applies well to your use case.

Besides that there is a jQuery-File-Upload server-side implementation in pyramid. But from my point of view it is of limited use and is currently not running without some fixes.

Nevertheless a WSGI middleware like gp.fileupload plugged into your WSGI pipeline is able to do max_file_size checks.

Your second problem is about chunked file uploads. This increases complexity in your server-side upload handler implemented in pyramid, but enables you to create hashes from large files with less memory.

Community
  • 1
  • 1
Sascha Gottfried
  • 3,303
  • 20
  • 30
  • client side validation for file size limits and allowed mime types are useful features offered by jQuery-File-Upload in purpose. Given this you can reject upload at client side. What is the benefit of choosing a jQuery plugin for file upload and not using any of its advanced features? – Sascha Gottfried Jan 27 '14 at 21:30
  • Another jquery plugin that talks about increased user experience if client side validation is applied - http://adamsanderson.github.io/jQuery-File-Validator/ – Sascha Gottfried Jan 27 '14 at 21:33
  • Client side validation could save some resources and improve user experience, but it's hardly a valid security measure – Riccardo Cagnasso Jan 29 '14 at 02:15
  • I always related my recommendation to your use case. Your specific question is not really about security, it is about memory and storage constraints. Instead of teaching me, go on with your application. You have a strange way to thank people giving you hints for your problem. – Sascha Gottfried Jan 29 '14 at 10:20
  • If you would really relate your recommendation to my use case, you would have considered what i said about my need to hash the file while is uploaded. Unless you are suggesting to hash it client-side. Now THAT would be a security issue for sure. I'm neither thanking you nor insulting you, i'm just explaining why your answer is not good for me. – Riccardo Cagnasso Jan 29 '14 at 13:07
  • Oh, well, for some reason i missed the last part :/ i'm sorry +1 to you. Yeah, chunked file uploads + client side validation + post limitation on frontend web server might work well, even if i don't really like the complexity involved – Riccardo Cagnasso Jan 29 '14 at 13:16
0

Maybe you can find some help in this project:

https://pypi.python.org/pypi/gp.fileupload/

Tiberiu Ichim
  • 641
  • 7
  • 7
  • It really seemed the solution... but it isn't. Even in this library all the checking is done after the file is uploaded. If i upload a 200M file with a 5 M limit, it uploads the whole file and then it warns me that it was too big. Disappointing. I'll try with websockets – Riccardo Cagnasso Jan 26 '14 at 15:47