3

webapp2_extras' sessions are pretty useful, and I've been using them to keep users logged in and such. But there's not much documentation about it. For instance: which is the memory limit for an entry in the session dictionary?

So far I've stored strings and numbers. But I need to store a whole image, uploaded from a HTML form (the idea is that this image is temporary and it may be discarded, so I don't want to store it in the datastore until I'm sure I have to do it). It doesn't seem to work, and I guess I'm hitting a memory problem:

self.session['photo_image']  = self.request.get("photo_image")

Is that so? I guess there are memory limits, but I can't find them. How else could I store a temporary image in GAE?

Notnasiul
  • 236
  • 1
  • 2
  • 12

1 Answers1

3

You can store it in 'instance memory', e.g. create a global list when your script starts up and append incoming images to that, then remove it once done. Of course, you'll soon run of of memory there also if you have lots of users/large files. And you'll lose it all when the instance shuts down and you'll (might) have problems if more then one instance is running.

So Memcache sounds perfect for this. using Memcache

 def get_data():
    data = memcache.get('key')

    if data is not None:
        return data
    else:
        data = self.query_for_data()
        memcache.add('key', data, 60)
        return data

Yes, it's not 100% reliable as I'm sure you've heard but if you are not using 100's of files that are huge and keeping them round for ages you probably won't have issues. As it's first in first out (IIRC) if you process them more or less in order that'll be even less likely to lose data. I think memcache is going to be your best option here, try it.

Paul Collingwood
  • 9,053
  • 3
  • 23
  • 36
  • My current solution consists on adding a 'temporal_image' field to the User model: it contains the image currently being uploaded by each user. But it requires writing to and reading from the datastore, which I'm trying to avoid. So I'll try your solution first. Thanks! – Notnasiul Nov 26 '12 at 13:57
  • Agh, wait: "The combined size of the serialized key and value must be at most 1 megabyte." So no, memcache is not an option, I fear :( – Notnasiul Nov 26 '12 at 14:07
  • 1
    That's per item, not total don't forget! And you can split it up into pieces as described here: http://stackoverflow.com/questions/9127982/avoiding-memcache-1m-limit-of-values so it's no so bad really. – Paul Collingwood Nov 26 '12 at 14:09
  • Right, right, I know it's per item. But these are pretty big images (more than 1Mb, at least, I fear). And right, I could break them in smaller pieces, but it seems that things are even less reliable when doing so... I think I should stick to the datastore X( – Notnasiul Nov 26 '12 at 14:23
  • TBH you should try it first and see exactly how unreliable it is. I've used memcache to do 1/4 million transactions in a day no problem. It might be unreliable if you are using it as a replacement for the datastore but if you are using it as intended then you should be a-ok! but as you prefer! – Paul Collingwood Nov 26 '12 at 14:28
  • You are right. Anyway, it seems I managed to compress my data, so I may be easily under the 1Mb mark! Thank you ;D – Notnasiul Nov 26 '12 at 14:50