Google App Engine java OutOfMemoryError when putAll on memcache

Question

I'm storing a map of around 20k entries on GAE memchace. Each entry it's around 1Kb big. I'm getting the following error. Is there a limitation? In my understanding the limitation of 1Mb it's for every entity you put in the memcache not the whole batch.

java.lang.OutOfMemoryError: Java heap space
at com.google.appengine.repackaged.com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:34)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper.makeAsyncCall(MemcacheServiceApiHelper.java:104)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl.doPutAll(AsyncMemcacheServiceImpl.java:521)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl.putAll(AsyncMemcacheServiceImpl.java:564)
at com.google.appengine.api.memcache.MemcacheServiceImpl.putAll(MemcacheServiceImpl.java:112)
[...]

And my code looks like:

final HashMap<EntityToStoreKey, EntityToStore> map = new HashMap<EntityToStoreKey, EntityToStore>();
for (EntityToStore entityToStore : entitiesToStore) {
    index.add(entityToStore);
    map.put(EntityToStoreKey.key(entityToStore.getId(), false), entityToStore);
}
entityToStoreCache.putAll(map, portalCacheTimeout);

And the issue only happens on a prod environment, not in local.

Any help/tip ?

This doesn't appear to be a `memcache` issue. Before the data is fed to memcache, it is being serialized, and that is when you're exceeding your heap's size. If your entities are 1k in size, this is surprising. Could there be a cyclical reference within the entitiy? — cheeken, Apr 01 '12 at 19:42

score 2 · Answer 1 · answered Apr 01 '12 at 19:36

The size of memcache is a total black box in GAE. Don't trust it to be anything. You are correct, no single entity can be larger than 1 MB but you have NO IDEA how big your total memcache is nor do you have any idea what the eviction policy is.

The latest release of GAE seems to be giving us some visibility into memcache but up until now it's really been a black box. I recommend not doing what you're trying to do. If you need to warm your cache do it in a loop and only load the most relevant content.

score 1 · Accepted Answer · edited May 23 '17 at 12:25

Are you using Appstats? I've found Appstats uses a large chunk of memory, see my question here: How to reduce the memory usage of Appstats on Google App Engine Java

I've experienced OutOfMemoryError when trying to load around 20,000 entries at once, similar size as yours. Now I load them in batches, 1000 at the time. Here's an example code using Siena (but it should be easy to port to other ORMs): http://groups.google.com/group/siena-discuss/msg/cd874589ef7aaa66

I find handy to use the method List.subList() followed by List.remove() or List.clear()

[Update]

I've found an undocumented parameter that limits the size of StackTrace:

<filter>
  <filter-name>appstats</filter-name>
    <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
    <init-param>
      <param-name>maxLinesOfStackTrace</param-name>
      <param-value>16</param-value>
    </init-param>
</filter>

score 1 · Answer 3 · answered Apr 02 '12 at 11:39

1

You're getting an exception because your Java servlet container is running out of memory, not because of anything related to memcache.

If you're trying to set 20,000 memcache keys in a single request, though, I suspect you're Doing It Wrong.

answered Apr 02 '12 at 11:39

Nick Johnson

100,655
16
128
198

20k entities of 1kb makes it 20Mb max, I don't think this is that big. Is it ? Can't I have it all in the memcache ? – Jordi P.S. Apr 02 '12 at 15:23
@planadecu Sure, you can, but you're still talking about writing _20,000_ records in a single request. It's not just about the total size. – Nick Johnson Apr 02 '12 at 16:18

Google App Engine java OutOfMemoryError when putAll on memcache

3 Answers3