understanding memory usage and Frontend Instance Classes on GAEJ

Question

I'm trying to find the best way to configure my App instances on GAE.

I run a Saas on GWT / GAEJ. I have a small number of power-users who use the app intensively all day (rather than a large number of users who use the app for a few minutes every day).

As such I'm trying to find the best way to configure my use of Frontend Instance Classes to make it the most efficient, and give the best user experience.

I have enabled billing, and found that I need to have idle instances running to avoid latency when instances are started. I use JDO, and the when started each instances takes a long time to initialize the datastore access. So I start a number of app instances and have them running in idle mode. This gives a great user experience, but clearly means I pay for instances that just sit idle - not ideal.

I will investigate whether I can do that more efficiently.

However this is background, my real questions are as follows; If I look at the memory usage for my instances they regularly say 136MB etc (they start at around 66MB). So I imagine I have some memory leaks to find. But specifically I'd like to know:

1. I also use Memcache, presumably this memory is taken into account in the above calculation?

2. I currently use F1 instance classes, which have a memory size of 128MB. So what does it mean for my instances which seem to settle into around 136MB in size? Will they be running much more slowly as they'll be swapping out to disk the whole time? Would I be better off running a single F2 instance instead of 2 F1 instances for this reason?

3. I have found very annoyingly that GAE starts new instances even though I have 2 idle instances. This is despite me setting the min latency very high (7.5secs). I did read in the docs that this setting will have little effect when I use idle instances, but how then do I ensure that only idle instances are used, without resorting to starting new (which always results in greater latency due to the datastore initialization problems mentioned above)? (and increased cost to me by way of more instance-time)

Am I mis-understanding something? many thanks for any help.

Answer 1

Few notes:

a. JVM acquires the memory on-demand, but rarely releases it back to OS. This is not a memory leak.

b. You can tweak number of pending instances and response times. This gives you some control over the number of running instances. See the docs on performance settings.

c. If your code is slow to start and you don't want long response times during startup, you might want to use warmup requests.

Now on to questions:

1. Memcache is a service external to the frontend instances and AFAIK does not store objects on the frontend instance. So the total number of objects in Memcache should not reflect on the memory used by frontend instance.

2. The actual underlying implementation of frontend instances is not known (outside Google at least), but I doubt they use local disks for swap. You can always try to use F2 instances and see how it works out.

3. Spin up of new instances is commanded by latency settings. If a request sits in the queue too long a new instance will be spun-up. If you instances take too long to spin up this might create a vicious circle. Try using warmup requests.

Answer 2

I also had the Problem that JDO / DataNucleus initialization made the first request to an instance very slow. I use MemCache to cache almost everything that is in the Datastore. Preloading everything (for now, when data grows I will change that to preloading important things and only touch not so important objects to initialize JDO) in the warmup request solved the issue for me, that the 1 resident instance that I have only got a few request. Now the usage of my 1 resident and the other on demand instances is much better distributed - still the resident instance only gets one-third of the requests.

I think that the scheduler gets a problem with the latency calculation when the first request(s) take longer than later ones.