1

I'd like to load lots of data from a Google Datastore table. For performance, I'd like to run, in parallel, a few queries that each loads a lot of objects. Cursors are not suitable for the parallel execution.

QuerySplitter is. However, for QuerySplitter you have to tell it how many splits you want, and what I care about is loading a certain number of objects. The number is chosen for the needs of my application, large but not not too large, say 800 objects. It's OK if the number of objects returned by each query is only very roughly the same; nothing worse would happen that different threads running different amounts of time.

How do I do this? I could query all objects keys-only in order to count them, and divide by 800. Is there a better way.

Joshua Fox
  • 18,704
  • 23
  • 87
  • 147
  • Rather than querying (even keys only) for all entities of a kind, you could use the [Datastore Statistics API](https://cloud.google.com/datastore/docs/concepts/stats) to get `__Stat_Kind__` – tx802 Apr 18 '16 at 13:43
  • Thank you! I did not know about that and that is probably what I will do to load an entire table. However, we'd like the capacity to load part of the data in a table, based on a filter -- though still a huge amount of data. I have edited the question accordingly. If you'd like to put your comment into an answer, I can upvote and and/or accept it. – Joshua Fox Apr 18 '16 at 14:13

1 Answers1

1

Querying all your entities (even keys only) might not scale so well, but you could run your query/ies periodically and save the counts in datastore or memcache, depending on how frequently you need to run your job.

However, to find all the entities of a given kind you can use the Datastore Statistics API which should be a lot quicker. I don't know how frequently the stats are updated but it's probably the same as the stats in the console.

If you are going to more frequent counts, or figures for filtered queries, you might consider sharded counters. Since you only need an approximate number, you could update them asynchronously on each new put.

tx802
  • 3,524
  • 2
  • 17
  • 22
  • 1
    For future reference: Sharding counters documented here https://cloud.google.com/appengine/articles/sharding_counters?csw=1 Discussion of count operations here http://stackoverflow.com/questions/421751/whats-the-best-way-to-count-results-in-gql – Joshua Fox Apr 21 '16 at 06:11