I'm trying to implement the code that Guido posted as an answer to the question Avoiding Memcache 1M limit of values. Everything seems(?) to be working when I first load the page and add values to memcache, but when I re-load it and retrieve values from the memcache I get a strange error:
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 194, in Handle
for chunk in result:
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1036, in appstats_wsgi_wrapper
result = app(environ, appstats_start_response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/appengineurlhere/mypythoncode.py", line 87, in get
entitylist = retrieve("entitylist")
File "/appengineurlhere/mypythoncode.py", line 53, in retrieve
return pickle.loads(serialized)
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
KeyError: ':'
Here's the store/retrieve code:
def store(key, value, chunksize=750000):
serialized = pickle.dumps(value, 2)
values = {}
for i in xrange(0, len(serialized), chunksize):
values['%s.%s' % (key, i//chunksize)] = serialized[i : i+chunksize]
memcache.set_multi(values)
def retrieve(key):
result = memcache.get_multi(['%s.%s' % (key, i) for i in xrange(32)])
serialized = ''.join([v for v in result.values() if v is not None])
return pickle.loads(serialized)
And this is how I'm using it:
try:
entitylist = retrieve("entitylist")
except EOFError:
entitylist = MyModel.all().fetch(None)
store("entitylist", entitylist)
Other oddities:
- I don't see this on my development server; just the production App Engine site. (Though the data on my development server is slightly different; I believe it all fits into the standard 1MB memcache size, where the production data is larger.)
- When I search for "entitylist" on both my development and production admin panel, App Engine tells me "no such key". Yet based on the total size of the memcache shown (this is the only place I've implemented memcache), it absolutely appears that something is getting cached.
Can anyone please point me in the direction of something I should look at or fix?
