5

Appengine docs mention a 1Mb limit on both entity size and batch get requests (db.get()): http://code.google.com/appengine/docs/python/datastore/overview.html

Is there also a limit on the total size of all entities returned by a query for a single fetch() call?

Example query:

db.Model.all().fetch(1000)

Update: As of 1.4.0 batch get limits have been removed!

  • Size and quantity limits on datastore batch get/put/delete operations have been removed. Individual entities are still limited to 1 MB, but your app may batch as many entities together for get/put/delete calls as the overall datastore deadline will allow for.
csytan
  • 188
  • 2
  • 12

3 Answers3

6

Theres no longer a limit on the number of entities that can be returned by a query, but the same entity size limit applies when you are actually retrieving / iterating over the entities. This will only be on a single entity at a time though; it is not a limit on the total size of all entities returned by the query.

Bottom line: as long as you don't have a single entity that is > 1Mb you should be OK with queries.

Karl Rosaen
  • 4,508
  • 2
  • 27
  • 30
  • Thanks Karl. Would you happen to know the size limit on the total size of entities for a query (e.g. for a single fetch())? – csytan Nov 10 '10 at 01:49
  • 1
    I don't think there is one; i think the only limit is in the timeout of the entire request that you would run into if you tried iterating over the results of a query that had a million results. – Karl Rosaen Nov 10 '10 at 01:53
3

I tried it out on production and you can indeed exceed 1 Mb total for a query. I stopped testing at around 20 Mb total response size.

from app import models

# generate 1Mb string
a = 'a'
while len(a) < 1000000:
    a += 'a'

# text is a db.TextProperty()
c = models.Comment(text=a)
c.put()


for c in models.Comment.all().fetch(100):
    print c

Output:

<app.models.Comment object at 0xa98f8a68a482e9f8>
<app.models.Comment object at 0xa98f8a68a482e9b8>
<app.models.Comment object at 0xa98f8a68a482ea78>
<app.models.Comment object at 0xa98f8a68a482ea38>
....
csytan
  • 188
  • 2
  • 12
2

Yes there is a size limit; the quotas and limits section explicitly states there is a 1 megabyte limit to db API calls.

You will not be able to db.get(list_of_keys) if the total size of the entities in the batch is over 1 megabyte. Likewise, you will not be able to put a batch if the total size of the entities in the batch is over 1 megabyte.

The 1,000 entity limit has been removed, but (at present) you will need to ensure the total size of your batches is less than 1 megabyte yourself.

Robert Kluin
  • 8,282
  • 25
  • 21
  • 1
    While your information is correct, I should clarify that the question is about the limit for *queries* and not batch fetch. – csytan Nov 10 '10 at 07:24