1

To upload a file from url to a cloud storage system, usually it is required to download that file on a server, and then upload it to the cloud storage.

For large files, it may be required to write the file on disk instead of memory. Since app engine does not support writing on disks, are there any other options for doing this on app engine?

I understand that managed vm is an option, but I'm trying to make sure that it's definitely not possible to do this on classic app engine.

ali
  • 531
  • 5
  • 22
  • you can copy from one stream to another, why you want to write to a disk? – Igor Artamonov Oct 06 '15 at 14:47
  • Sure, but does app engine client libraries support this? Is it possible to pipe `urlfetch` to `cloudstorage` python packages for streaming? Could you provide a code sample as an answer? – ali Oct 06 '15 at 15:12
  • oh, I don't know about python libs, but java libs have streams and everything. you could probably make a java module for this task – Igor Artamonov Oct 06 '15 at 18:33

2 Answers2

2

To overcome the lack of local disk on GAE you can:

To download the files to GAE you could use the URL service. But there are 2 limitations to keep an eye on:

If the server offering the downloads supports multi-part downloads it might be possible to get a solution working for any file sizes with this info.

Note: this is just theoretical, I've only thought about this, I didn't actually tried it.

Community
  • 1
  • 1
Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
0

Yes, it's possible. You can generate an upload URL (using Cloud Storage API) and provide it to a client that should use it in a POST request.

Here is docs for PHP but this approach should be working for Python as well.

Here is an example: https://github.com/GoogleCloudPlatform/storage-signedurls-python

Alexander Trakhimenok
  • 6,019
  • 2
  • 27
  • 52
  • How can this be used to stream a file from a url to the cloud storage upload url? – ali Oct 06 '15 at 17:04
  • It was not too clear from your question you are fetching/downloading file. I was thinking about situation when you want a user or 3d party app to upload a file. In you case my answer is not relevant. – Alexander Trakhimenok Oct 06 '15 at 17:07