2

If I want to get a property (no filtering) of 50,000 entities and display them all on one page, what would be the most efficient way to do that?

Should I fetch() then in pieces of 1000 at time or use the query iterator?

The entities themselves are small, having one or two StringProperty (unindexed, because no filtering is required), which are < 50 bytes.

PS: I want to display all 50k at once (ie. without the need for a "next" link)

Chad
  • 2,365
  • 6
  • 26
  • 37
  • 1
    50k is going to take a long time (possibly more then 30s) and cost a whopping 3.5 cents (multiply this with a number of such requests). You should seriously rethink your architecture. – Peter Knego Mar 25 '13 at 22:03
  • @PeterKnego I'm using NDB so shouldn't solve the cost problem (with its automatic cacheing)? Also, which takes the most time, clientside rendering or datastore fetching? – Chad Mar 25 '13 at 22:10
  • 1. Queries are not cached. 2. For querying 50k entities you'd need to do about 50 queries which take time. I suggest you try it. – Peter Knego Mar 25 '13 at 22:23
  • Yes, I suppose i'll have to try. By the way, the 1k entities per query limit has been removed if that's why you said 50 queries are needed: http://stackoverflow.com/questions/264154/google-appengine-how-to-fetch-more-than-1000 – Chad Mar 25 '13 at 22:32
  • Datastore will never return all results in one go. Iterator is doing lazy-loading behind the scenes, so it'll definitely be making a series of network roundtrips. Also see this (Guido, 2nd answer, is the author od NDB): http://stackoverflow.com/questions/10968439/what-is-the-google-appengine-ndb-gql-query-max-limit – Peter Knego Mar 25 '13 at 22:41
  • Thanks for that reference. I'll take Guido's advice and do benchmarks of different variations to find the best way to go. Thanks for your help. – Chad Mar 25 '13 at 22:46
  • Do you really need to see 50K items in the list at immediately - In most cases you couldn't actually see all of them on a single page without scrolling. Maybe a light weight javascript infinite scroller/list might work, only populating the visible parts as scrolled (say several hundred at a time). See https://github.com/enyojs/enyo/wiki/Lists for a type of js thing that will do this sort of thing. – Tim Hoffman Mar 26 '13 at 00:46

1 Answers1

1

If you regularly load 50k entities, you'd better join them all together and save to the BlobStore, so you can easily get them (or serve them) all at once. I don't see any point in storing them to the database, especially because you're not doing any filtering or sorting.

lenik
  • 23,228
  • 4
  • 34
  • 43
  • You know I have actually considered doing that from the beginning. I think I would have eventually went this route anyway but you get credit for helping me makeup my mind. ;) – Chad Mar 26 '13 at 04:02