4

It would really be nice if the WSAPI allowed for a larger maximum page size. There are many instances where I am pulling several thousand work items, and I am only in need of one field in the record. Having to pull the records 200 at a time means long wait times for the user while hundreds of individual pages are loaded.

The LBAPI page size should be some indication that this is certainly possible. It's max page size (if I remember correctly) is 10,000. I realize it's returning less information per record (OIDs instead of full objects) but with the optimizations made to the WSAPI in regards to bandwidth (not returning full collection detail) it seems like 200 is awfully low.

Charles Ferentchak
  • 1,237
  • 1
  • 9
  • 14
Conner Reeves
  • 1,381
  • 8
  • 13
  • Have you used `limit: Infinity` inside the WsapiDataStore config? – Blundering Philosopher Aug 24 '13 at 18:57
  • I have. The results are still loaded one page at a time though. If you look at the network activity you can see each individual request being processed sequentially. The fastest way I've found to process a large query with WSAPI is to find out out many pages there are going to be, and then run "loadPage(##)" for each page of the store (running all the queries at once) and group the records together once the've all returned. This can cause some problems when we're talking about 150+ requests being processed at once though. haha – Conner Reeves Aug 24 '13 at 21:25
  • I have been able to use the Infinity limit with success on several of my projects. (2.0rc1) I'm loading around 1,000 entries so I might not be pulling the volume of data you are. – amcolosk Aug 26 '13 at 15:14

1 Answers1

1

There are 2 problems here which are contributing to the slowness here. The first is the small page size. There are internal discussions happening right now about bumping this to 1000 (especially since now in v2.0 collections aren't returned in the request).

The second is that right now WsapiDataStore does its paging serially. The first page always has to be loaded serially, but after that the remaining n-1 should be able to be done in parallel.

There is a defect tracking the second item that will hopefully be addressed soon. Hopefully the first item will get some attention as well.

Kyle Morse
  • 8,390
  • 2
  • 15
  • 16
  • Thanks for the reply Kyle. So, it seems like the pros of increasing the page size are pretty apparent, but what are the cons that are keeping you guys from increasing it to 1000, or more? – Conner Reeves Aug 26 '13 at 21:52
  • We don't really know yet. It's mostly a prioritization/allocation problem. We need to model that change in our load testing environment and measure various scenarios. – Kyle Morse Aug 27 '13 at 14:50