0

I have a ~2MB file that my Google AppEngine server must use (not serve) as part of a computation for a service request.

That is, a client makes a particular request, my GAE server must first get the data from this ~2MB file, do some computations using this data, then serve a small response back to the client.

Where best do I store this data so that it can be quickly read and used by the server in the computation?

aez
  • 2,406
  • 2
  • 26
  • 46

1 Answers1

1

If the following assumptions hold true

  • the file is not going to require updates outside of appengine code updates
  • that the file is read only

Then deploy the file with your code and read the file into memory during startup (ideally using warmup requests) and just operate on it from memory. If you code has to have file based semantics to access the data (read,seek, etc) then read the file contents and wrap it in StringIO.

You will need to assign the value read from the file to a module level variable, that way whenever you get a new request you can just get the files contents by importing the module and referencing the name. ie. mymodule.filecontents

Tim Hoffman
  • 12,976
  • 1
  • 17
  • 29
  • Thanks Tim. Can you please clarify, how do I deploy the file with my code and how do I read it into memory during startup? And sorry, what is a warmup request? Thanks. – aez Jun 15 '12 at 10:30
  • ok, DO you understand how applications are deployed to appengine. If so the file is just included with the actual code. (You haven't said if your using python or java. I only work with python so anything more I say will be based on python. Just think of the file as another piece of code. The you can use standard open("path to file").read() type calls to read it into memory. – Tim Hoffman Jun 15 '12 at 10:37
  • And you can have a look at https://developers.google.com/appengine/docs/adminconsole/instances#Warmup_Requests for detail on warmup requests. Basically appengine will fire off a request to /_ah/warmup to startup an instance before sending traffic to it. By using the warmup request to read the file into memory you can then just read it once on startup, and not incur the read cost on every request. – Tim Hoffman Jun 15 '12 at 10:39
  • Thanks Tim. Yes, I've deployed a few test apps, but I don't know _where_ to put this file (and then what its path will be), and I'm wondering what you mean by "startup" since this file is used by the server only. But startup/warmup sounds interesting because I'm worried about having to read this large file for every client request. (edit--great info on "warmup", your comment post came just before I entered this comment) – aez Jun 15 '12 at 10:44
  • Are you using python ? If so have a look at this stackoverflow question about determining filepaths in appengine. http://stackoverflow.com/questions/61894/whats-a-good-way-to-find-relative-paths-in-google-app-engine – Tim Hoffman Jun 15 '12 at 11:26
  • I am using Java, but I will look into your link – aez Jun 15 '12 at 12:08
  • If you can just confirm where the file should go (in the war directory?) and then the relative path to read it in Java, I can accept your answer. – aez Jun 15 '12 at 13:21
  • Unfortunately I have never done anything with java on gae. However this stackoverflow discussion does look at how to do it with java http://stackoverflow.com/questions/4340653/file-path-to-resource-in-our-war-web-inf-folder – Tim Hoffman Jun 15 '12 at 13:24