4

I'm updating my project to use Azure.Data.Tables 12.6.1, and I can't figure out where to specify a Take value to limit the number of entities returned from a query.

In other words, I want to do something like this:

var limit = 150;
var results = table.QueryAsync<T>(limit);
await foreach (var page in results.AsPages().ConfigureAwait(false)) {
    // Regardless of how the server pages the results, 
    // only the top [limit] items are returned.
}

In the old API, you could set a Take property on the query object. How do I do this in the new API?

Brian Rak
  • 4,912
  • 6
  • 34
  • 44
  • Does the `MaxPerPage` parameter help you? – Skin Oct 16 '22 at 02:12
  • @Skin I don't think that does exactly what I want. If you know the results you want will be on the first page, then yes, it does. But let's say that the server is sending 100 entities per page and you want 120. Saying "120 max per page, please" doesn't change anything. The server still sends 100 per page and you needlessly retrieve 80 entities (because you have to retrieve two pages to get the 120 you need). I just want to say: regardless of the number of pages, give me 120, total. Or if I could say how many I want, max, in the NEXT page, that would be okay too. – Brian Rak Oct 16 '22 at 15:14

2 Answers2

2

As @Skin points out, the current SDK does not expose an explicit API for Take, but that was an intentional decision to ensure that it is clearer to developers what is really happening from the service perspective.

The old SDK supported a full IQueryable API, which made it easy to create very expensive queries that performed filtering client side after fetching the whole table from the service.

Although Take doesn't have the same problems as other Linq methods, the service doesn't really support it. It can only limit the number of results for a paged result (which is capped at 1000 by the service).

While I agree it is not as simple as a Take API, the current API makes it fairly straightforward to implement the equivalent functionality, while not hiding the fact that you may actually fetch more than your Take limit from the service.

This sample demonstrates how to iterate through the pages with a max items per page set.

Christopher Scott
  • 2,676
  • 2
  • 26
  • 26
  • Thank you for the reply, Christopher. I have two questions. This page (https://learn.microsoft.com/en-us/rest/api/storageservices/querying-tables-and-entities) shows that $top is a supported parameter, allowing you to limit the number of entities fetched. Isn't that what Take was utilizing? – Brian Rak Nov 01 '22 at 11:18
  • And secondly, the same link shows that you are not guaranteed to get the full 1000 possible entities per page if the result set crosses a partition boundary. So let's say I have 500 entities spread across 5 partitions, and I want the first 120 of them. Without Take or $top, there is no way for me to get only the first 120. I am forced to fetch 200 and then discard 80. Or am I not seeing something here? – Brian Rak Nov 01 '22 at 11:22
  • 1
    Ah, I just found a pertinent line (again from the above link): "Note that the $top query option in the case where results are paginated specifies the maximum number of results per page, not the maximum number of results in the whole response set." So I guess this is telling me that there never was any server-side Take in the way that I understood it (or perhaps there was before and it was deprecated there too). So, just to confirm, in my scenario in the previous comment, am I correct to understand that I should just fetch until I have enough and discard the ones I don't need? – Brian Rak Nov 01 '22 at 11:27
  • 1
    Exactly - $top doesn't really exist. So fetching at the appropriate max-per-page granularity until you get the amount you want the best you can do. – Christopher Scott Nov 01 '22 at 20:16
  • 1
    Okay, I grudgingly accept your answer. :-) Seriously, though, it would be really useful to be able to specify a max number of entities to return, across whatever number of pages is required to get it. I hope you'll consider that for the future! Either way, thank you for your time here getting this sorted out. – Brian Rak Nov 02 '22 at 05:20
1

This may be a little controversial but I'm going to add this as an answer ... it looks like it's been raised as a feature request only a number of weeks ago and it's now been added to the backlog ...

https://github.com/Azure/azure-sdk-for-net/issues/30985

You're not the only one who has the same requirement.

Skin
  • 9,085
  • 2
  • 13
  • 29