Google AppEngine: how to count a database's entries beyond 1000?

Question

Duplicate of "how does one get a count of rows in a datastore model in google appengine?"

I want to know how many users I have. Previously, I achieved this with the following code:

users = UserStore.all()
user_count = users.count()

But now I have more than 1,000 users and this method continues to return 1,000.

Is there an efficient programmatic way of knowing how many users I have?

score 14 · Accepted Answer · edited Jul 23 '14 at 06:37

14

It is indeed a duplicate and the other post describes how to theoretically do it, but I'd like to stress that you should really not be doing counts this way. The reason being that BigTable by its distributed nature is really bad for aggregates. What you probably want to do is add a transactional counter to that entity, and if there are lots of transactions a sharded counter. See: http://code.google.com/appengine/articles/sharding_counters.html

UPDATE: Since 1.3.1 cursors make stuff like this a lot easier: http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Query_Cursors

edited Jul 23 '14 at 06:37

alex

479,566
201
878
984

answered Apr 28 '09 at 07:15

Koen Bok

3,234
3
29
42

2

+1000. Counting every user for every request is a really bad idea. – Nick Johnson Apr 28 '09 at 09:59
I ended up implementing this! Thanks a million, my performance improved a lot (: – Manuel Araoz Apr 30 '09 at 02:43
Another way of accomplishing this without having to implement sharding counters is by accessing the Datastore statistics. Here is a full explanation of 3 methods to count the number of entries for a specific type: https://blog.svpino.com/2015/03/08/how-to-count-all-entries-of-a-given-type-in-the-app-engine-datastore – svpino Mar 08 '15 at 16:14

score 2 · Answer 2 · answered Aug 23 '10 at 14:58

2

Since version 1.3.6 of the SDK the limit of 1000 on the count function has been removed. So a call to the count function will now return the exact number of entities, even if there are more than 1000. Only limitation would be if you had so many entities that the count function would not return before the request has a timeout.

answered Aug 23 '10 at 14:58

Tom van Enckevort

4,170
1
25
40

Instead of using count(), how about using the len()? Check here: http://alwaysthecritic.typepad.com/atc/2009/02/google-app-engine-query-and-querycount.html – DocWiki Jul 05 '11 at 17:47
That method was from the time it still had the limit of 1000. – Tom van Enckevort Jul 06 '11 at 08:37

score 2 · Answer 3 · answered Apr 28 '09 at 01:00

2

Use pagination like these examples here.

answered Apr 28 '09 at 01:00

Bjorn

69,215
39
136
164

score 0 · Answer 4 · answered Oct 11 '12 at 17:32

0

For Python GAE SDK, you can increase the argument "limit" of the count method: https://developers.google.com/appengine/docs/python/datastore/queryclass#Query_count

answered Oct 11 '12 at 17:32

Paulo Cheque

2,797
21
22

score 0 · Answer 5 · edited Jul 23 '14 at 06:36

0

I have write this method to count a query, but how said Nick Johnson maybe it's a bad idea...

def query_counter (q, cursor=None, limit=500):
  if cursor:
      q.with_cursor (cursor)
  count = q.count (limit=limit)
  if count == limit:
      return count + query_counter (q, q.cursor (), limit=limit)
  return count

edited Jul 23 '14 at 06:36

alex

479,566
201
878
984

answered Aug 23 '10 at 12:11

sahid

2,570
19
25

Google AppEngine: how to count a database's entries beyond 1000?

5 Answers5

Linked

Related