2

I'm creating an API for a Django application and one of the PUT requests allows for a (very) large file upload. I'm trying to host the application on Heroku, but I'm running into some issues with the file upload and 30 second limit and ephemeral filesystem. I'm now trying to pass the file off onto an s3 bucket (via boto and abusing the multipart_upload function), but the process is dying before the upload can complete. I found this extremely helpful answer early on, which was excellent advice for the API in general, but doesn't solve the Heroku issue.

In my view

for i, handler in enumerate(upload_handlers):
    chunk = request.read(handler.chunk_size)
        while chunk:
            handler.write_multipart_upload(chunk, counters[i], filename=file_name)
            counters[i] += len(chunk)
            chunk = request.read(handler.chunk_size)

for i, handler in enumerate(upload_handlers):
    file_obj = handler.file_complete(counters[i])

In my handler

def write_multipart_upload(self, raw_data, start, filename=None):
    if not self.mp:
        self._validate_file(filename=filename)
        self.mp = self.bucket.initiate_multipart_upload('feeds/' + self.file_name)

    self.buffer.write(raw_data)

    if self.buffer.len:
        self.buffer.seek(0)
        self.mp.upload_part_from_file(self.buffer, self.total_chunks)
        self.buffer.close()
        self.buffer = StringIO()

def file_complete(self, file_size):
    self.mp.complete_upload()
    self.file

Maybe this has become too complicated. At this point, I'm looking for something simple and maintainable. If anyone has a solidly-working example of a PUT-based API for large file uploads, I'll scrap Heroku and be happy with that. Thanks in advance.

Community
  • 1
  • 1
Jared
  • 21
  • 1
  • I can't answer specific for django but Amazon S3 supports direct upload so you can bypass Heroku altogether but then receive a message into your application when the upload completes - you might want to look into this as it avoids the 30s Heroku timeout. – John Beynon Jun 07 '12 at 21:09
  • I am also interested in that issue. Do you have some news about it? – Natim Mar 27 '13 at 10:24

0 Answers0