3

I've got a Django app that accepts uploads from jQuery uploadify, a jQ plugin that uses flash to upload files and give a progress bar.

Files under about 150k work, but bigger files always fail and almost always at around 192k (that's 3 chunks) completed, sometimes at around 160k. The Exception I get is below.

exceptions.IOError
request data read error
File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 171, in _get_post
    self._load_post_and_files()
File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 137, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META,     self.environ[\'wsgi.input\'])
File "/usr/lib/python2.4/site-packages/django/http/__init__.py", line 124, in parse_file_upload
    return parser.parse()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 192, in parse
    for chunk in field_stream:
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 314, in next
    output = self._producer.next()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 468, in next
    for bytes in stream:
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 314, in next
    output = self._producer.next()
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 375, in next
    data = self.flo.read(self.chunk_size)
File "/usr/lib/python2.4/site-packages/django/http/multipartparser.py", line 405, in read
    return self._file.read(num_bytes)

When running locally on the Django development server, big files work.
I've tried setting my FILE_UPLOAD_HANDLERS = ("django.core.files.uploadhandler.TemporaryFileUploadHandler",) in case it was the memory upload handler, but it made no difference.

Does anyone know how to fix this?

Jake
  • 12,713
  • 18
  • 66
  • 96

2 Answers2

2

The problem could be:

  • your internet connection (some firewall blocking large uploads, just try uploading to some other service to test that).
  • your webbrowser (try disabling flash to see if that works)
  • your webserver (perhaps your webserver prohibits large uploads?)

So... to be a bit more helpful you'll have to give us some info about which part is the problem (should be fairly easy to test) and if it's the webserver (which I'm guessing) what kind of webserver you're running, what kind of Python handler (mod_wsgi, mod_python, fastcgi, etc..)

Wolph
  • 78,177
  • 11
  • 137
  • 148
  • Just tested uploading a large file on another site with no problem. I'm running python 2.4, django 1.1, mod_wsgi. I'll try to test uploading without the flash uploader now. – Jake Apr 14 '10 at 23:27
  • I can upload to a different Django project on the same server using the Django admin interface (normal form file field). So perhaps it's a flash or uplodify issue, I don't know where to start to work that out. – Jake Apr 14 '10 at 23:43
  • Another update, I created a form posting to the same place as uploadify did, and I get the same error. So... Not my browser or internet connection, not my server, not uploadify. The main difference I can see between the project that works and the one that doesn't is that one uses django admin interface to upload to a models file field, and the broken one just puts downloads in a folder. But the error I have is a read error. – Jake Apr 15 '10 at 00:27
  • The "read error" means that the server has lost the connection to the client for some reason. So... during the upload the connection was somehow broken and the server can't read new data from the client anymore. I'm at a loss to explain it though, if it's not working with the normal upload form either... What browser, server and Python handler are you running? I'm guessing some bug in your webserver/webbrowser is biting you. – Wolph Apr 15 '10 at 01:47
  • I was using FF3/windows. In Chrome/windows it gets to a higher percentage before failing. In FF3/mac it works! I'm not sure what you mean by Python Handler, is that mod_wsgi/apache? I'm running centOS on a media temple DV server (python 2.4, django 1.1) – Jake Apr 15 '10 at 02:03
  • Yes, that's what I meant :) THere are several reports of uploading failing in certain situations with Django: http://stackoverflow.com/questions/411902/django-file-upload-failing-occasionally http://www.pubbs.net/200909/django/82114-ioerror-request-data-read-error-on-post-anyone-solved-this.html http://groups.google.com/group/django-users/msg/c625a04cb3bf9a8a Some say it's keep-alives in combination with Safari others have different opinions. But I don't see a real solution. – Wolph Apr 15 '10 at 08:19
2

This is common with MediaTemple -- I encountered this problem, fixed it, and somehow it's broken again. It has to deal with something about NFS locking, and the system calls used by django's storage modules.

... (10 minutes later) ...

Found it:

Anyone trying to run the latest 1.1 svn release will have noticed that the above fix no longer works since TemporaryFileUploaderHandler now uses the locking code also. A workaround until mediatemple and/or django devs figure this out is to edit django.core.files.locks and change the two calls to fcntl.lockf to fcntl.flock

Here's a relevant bug report for anyone who's interested: http://code.djangoproject.com/ticket/9400

http://osdir.com/ml/DjangoUsers/2009-06/msg01014.html -> https://forums.mediatemple.net/viewtopic.php?id=2514

Marcel Chastain
  • 2,101
  • 2
  • 18
  • 17