1

I like Google App Engine for the Datastore (and the fact that I can develop in Python) but they have 1MB limits on fetching and uploading files. On the other hand it wasn't made for that I guess.

I managed to create a service that is getting a URL (or a file from user), fetching it on GAE and then putting it on Google Storage. Works perfectly fine, but for files that are less than 1MB. There are workarounds of course, but I think it should be feasible to do it faster by skipping the GAE for data retrieval.

So my question is: Is it possible to keep my datastore on GAE and by knowing only the URL (or a special form to upload files) create the headers and then letting Google Storage to fetch the file and store it without the 1MB limit?

Community
  • 1
  • 1
Lipis
  • 21,388
  • 20
  • 94
  • 121

2 Answers2

1

No. Google Storage doesn't support fetching the file for you.

Nick Johnson
  • 100,655
  • 16
  • 128
  • 198
  • Oook...!! Then I guess we are switching to EC2 and since we are there then S3 is the way..! I really wanted to use GAE and GS but looks like it's not for this project :( – Lipis Nov 06 '10 at 12:27
  • Why not run a task that does the fetching and storing on EC2 or elsewhere, and the rest of the app on App Engine? It's not an all-or-nothing proposition. – Nick Johnson Nov 06 '10 at 14:14
  • well because the rest of the app is already somewhere else and I'm working on other stuff.. I was assigned with the media storing part and I wanted to do that service somewhere else to off load our servers... I'm a big fan of GAE.. and I was kinda disappointed yesterday when I faced all that, especially when everything worked :D but I'm not done with GAE and few personal project that we have in mind.. will be there!! Thanks for the help.. and do something about this 1MB and better integration with GS..!! – Lipis Nov 06 '10 at 19:52
1

It doesn't matter where the file is stored. Not only is there a 1MB datastore entity size limit, but there is a 1MB in-memory datastructure limit. So even if you stored the file somewhere else, you would have to split it into pieces to be able to operate on it on the app engine.

For Java on the GAE, I suggest looking at gaevfs (http://code.google.com/p/gaevfs/). It turns the app engine datastore into a filesystem that can store files of arbitrary size and operate on them just as normal files.

Travis Webb
  • 14,688
  • 7
  • 55
  • 109
  • we'll have a lot of data to store and it's either GS or S3.. it's too expensive to use the datastore for filestorage and having to deal with all these limitations.. – Lipis Nov 06 '10 at 19:46