1

I'm trying to send large files (50MB-2GB) that I have stored in S3 using filepicker.io to the Google App Engine Blobstore (in Python). I would like this to happen without going through a form on the client browser as it defeats the project requirements (and often hangs with very large files).

I tried multiple solutions, including:

  • trying to load the file into GAE with urlfetch (but GAE has a 32MB limit for requests/responses)
  • constructing a multi-part form in python and sending it to blobstore.create_upload_url() (can't transfer the file via url, and can't load it in the script because of the 32MB limit)
  • using boto to read the file straight into the blobstore (connection times out and logs show encountered HTTPException exception from boto that triggers CancelledError: The API call logservice.Flush() was explicitly cancelled. from GAE that crashes the process.

I am struggling to find a working solution. Any hints on how I could perform this transfer, or how to pass the file from s3 as a form attachment without loading it in python first (ie. just specifying its url) would be very much appreciated.

Community
  • 1
  • 1
ergelo
  • 923
  • 2
  • 9
  • 15
  • use the http libraries, not a full answer but check it out –  Apr 12 '13 at 12:04
  • @peterretief you mean something like [python requests](http://www.python-requests.org/)? – ergelo Apr 12 '13 at 13:43
  • I posted a question related to this suggestion [here](http://stackoverflow.com/questions/15973204/using-python-requests-to-bridge-a-file-without-loading-into-memory) – ergelo Apr 12 '13 at 14:09

1 Answers1

1

The BlobstoreUploadHandler isn't constrained by a 32MB limit: https://developers.google.com/appengine/docs/python/tools/webapp/blobstorehandlers. However, I'm not sure how this might fit into your app. If you can post the file to an endpoint handled by a BlobstoreUploadHandler, then you should be good to go.

Robert Moskal
  • 21,737
  • 8
  • 62
  • 86
  • thanks Robert, I have a BlobstoreUploadHandler already. My problem is how to send it the very large file that I have stored on S3 - I can't find a way to attach it to a POST request without loading it elsewhere in GAE first (which will cross the 32MB limit). Hope this makes sense. – ergelo Apr 12 '13 at 14:18
  • There are those backend processors that don't have any limitations at all. Have you looked into those? It is an interesting problem. I suppose you can't just leave the files in S3. If you are letting the user interact with the app through the browser then you can use the html5 File api: http://www.html5rocks.com/en/tutorials/file/dndfiles/#toc-slicing-files to load the file into the browser (what happens when you load a 2GB file though). Otherwise, you'll have to run a process outside the app engine to do all this. – Robert Moskal Apr 15 '13 at 03:00