Slow query time when using GCP Datastore with GCP Cloud Functions

Question

I deployed a GCP HTTP triggered Cloud Function that queries an Entity from GCP Datastore by the key. Here is the code:

const datastore = require('@google-cloud/datastore');

exports.helloWorld = function helloWorld(req, res) {
  const client = datastore();
  const key = client.key(['Person', 'harry']);

  client.get(key, function(err, entity) {
    res.status(200).send(JSON.stringify(entity));
  });
};

According to the logs this function takes ~1.6 seconds to complete when invoked. Repeat invocations are not any faster.

Removing the query and responding to the HTTP request takes ~0.5 seconds to complete, so it seems the query takes ~1.1 seconds to complete. For me this is unusably slow and it seems unlikely that this is the intended performance of GCP Datastore.

I thought that perhaps the DB and the Function are running in different regions but I am unable to check, the instructions given in the documentation is incorrect (https://cloud.google.com/datastore/docs/locations#location-r). The region is not displayed on the page for me.

What might be the problem with my setup here? I was expecting ~50ms for simple queries rather than ~1100ms.

Answer 1

In this question, you can find attached screenshot of Google's stack trace and see that the best time with Datastore and GCP is about 100 ms.

Honestly speaking, we worked on GCF with Datastore more than 3 months, and this time usually worth than 100 ms, about 200 - 400 ms per call. I had conversation with GCP support and can confirm, that currently they have troubles with that, something with requests routing and optimizations between GCF and Datastore only. I've collected several performance tests data sets through Yandex-Tank, and average requests latency where about 800ms to 7 seconds (about 4-5 Datastore serial requests).

After 3 months of development we moved to App Engine and found that Datastore behaves way faster in this environment, and average time is about 20-30 ms (4-5x time faster) per Datastore request.

I also noticed, that Datastore lookup time almost not depends on the amount of data it's running through. Is it 1 record or 1000 records, time will be almost same 20-30 ms. I believe that this time even better if we can look on the Datstore itself without any network communications extras.

Now we are adding redis as caching service to speed up all the requests. I guess it might work with GCF and Datastore as well, but I would not expect this would be a guaranteed solution. Therefore, consider use GCF as utility processing units instead of primary processing endpoint.