2

This query was working for me until recently. I now have 135 InstallationSummary documents in my RavenDB. Instead of getting the most recent by start time, it's mostly working, but the last couple, most recent documents aren't showing up from this query. Am I querying incorrectly? Is there a different way to do OrderByDescending and Take with RavenDB that I should be aware of? Is there a document number limit to what I can query correctly?

Note: I have debugged this, and the query indeed returns what we see in the grid. There is no transformation done between the time the query is run and what is shown in the UI.

IEnumerable<InstallationSummary> installationSummaries =
  QueryAndCacheEtags(session => session.Advanced.LuceneQuery<InstallationSummary>()
  .Include(x => x.ApplicationServerId)
  .Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
  .Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)
  .OrderByDescending(summary => summary.InstallationStart)
  .Take(numberToRetrieve)).Cast<InstallationSummary>().ToList();

This grid should show a few more rows in it with start times greater than 1/19/2012 6:33:51 PM:

enter image description here

Edit: I removed Take(numberToRetrieve) from the query, and I'm only getting 128 of the total 160 InstallationSummary documents. I can see all 160 in RavenDB Studio, but only 128 return from the query. 128... 128... power of 2... Did I hit some limit?

Okay, it looks like I did hit the limit of 128: http://www.blogcoward.com/archive/2010/05/21/RavenDB-and-a-brief-design-philosophy-discussion-with-Ayende.aspx http://codeofrob.com/archive/2010/05/12/ravendb-basic-usage-considerations.aspx

But why? I have a Take() method in there. How am I supposed to get the 50 most recent documents?

As a bit of a hack, the query below will at least show the most recent. It isn't exactly what I want, because I want the most recent 50, regardless of date. As long as there aren't more than 50 since the start date, this will at least show the most recent items.

using Raven.Client.Linq;

DateTime startDate = new DateTime(2012, 1, 18);

IEnumerable<InstallationSummary> installationSummaries =
QueryAndCacheEtags(session => session.Query<InstallationSummary>()
.Include(x => x.ApplicationServerId)
.Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
.Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)                        
.Where(x => x.InstallationStart > startDate)
.OrderByDescending(summary => summary.InstallationStart)                        
.Take(numberToRetrieve)
).Cast<InstallationSummary>().ToList();

I had to go from a LuceneQuery to just Query and I had to add the Where clause.

Bob Horn
  • 33,387
  • 34
  • 113
  • 219
  • In addition to that, what is QueryAndCacheEtags ? That seems _very_ strange, especially since RavenDB already does this for you – Ayende Rahien Jan 20 '12 at 11:07
  • QueryAndCacheEtags() executes the query and stores the etags in a cache for when objects are later saved. Since I don't keep a session open for the lifetime of an app, and the user could save something hours after he retrieved it, I store the etags for later use. It's the only way I knew how to make this work. I'm still learning Raven. – Bob Horn Jan 20 '12 at 12:34
  • What's the method signature of QueryAndCacheEtags() is it Expression> or just Func<..>? – Matt Warren Jan 26 '12 at 23:52
  • protected static IEnumerable QueryAndCacheEtags(Func> func) – Bob Horn Jan 27 '12 at 00:20
  • That's the problem then, you've changed the query form IQueryable to IEnumerable, so the OrderByDescending doesn't get sent to RavenDB. In effect the query you're asking ravenDB to do is just `session.Query()`. All the rest is done in-memory after it's pulled back the 128 docs (that have no ordering or filtering applied) – Matt Warren Jan 27 '12 at 10:14
  • So change the method signature to `IQueryable QueryAndCacheEtags(Expression>> expFunc)` and it should allow RavenDB to use the Where clause and the OrderbyDescending parts of the query as well – Matt Warren Jan 27 '12 at 10:15
  • I'd recommend using Fiddler or looking at the RavenDB server console log so you can see the query that is being performed on the server. Then this will all make a bit more sense. – Matt Warren Jan 27 '12 at 10:16
  • Excellent suggestion, Matt. I just did what you suggested and it seems to be working correctly. If you want to post that as an answer, I'll accept it. I didn't use Expression<> though. Is that necessary? – Bob Horn Jan 27 '12 at 16:14
  • If you don't use expression the query becomes IEnumerable, which is not what you want. If you commen/uncomment out the line `QueryAndCacheEtags(..)` and hover your mouse over the rest of the function call, Intellisense will show you the difference. Anything that isn't IQueryable won't be done by RavenDB on the server. – Matt Warren Jan 29 '12 at 15:13
  • Thinking about it a bit more, not sure if it's actually the `IQueryable` bit that solved the problem in your case. Either way, glad you got it working – Matt Warren Jan 29 '12 at 15:23
  • I changed the signature of QueryAndCacheEtags to return IQueryable. Things seems to be working now, so I'm not sure if Expression is necessary. Anyway, I'll play with that and see. Thanks again. – Bob Horn Jan 29 '12 at 15:56

2 Answers2

3

Finally worked out the real issue.

IEnumerable<InstallationSummary> installationSummaries =
    QueryAndCacheEtags(session => session.Query<InstallationSummary>()
       .Include(x => x.ApplicationServerId)
       .Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
       .Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)                        
       .Where(x => x.InstallationStart > startDate)
       .OrderByDescending(summary => summary.InstallationStart)                        
       .Take(numberToRetrieve))
       .Cast<InstallationSummary>()
       .ToList();

The signature of the QueryAndCacheEtags(..) function is Func<T>, not Expression<Func<T>>. And it returns IEnumerable<T> not IQueryable<T>

This converts the statement from IQueryable<T> to IEnumerable<T> at that point. This means that the RavenDB server only processes the first part of the query, which has no filtering or ordering.

The rest of the statements and then applied in-memory to the 128 items you get back. Hence why you aren't seeing the item ordered or filtered properly.

There's a bit more info here and here

Generally you don't have to worry about the difference between Func<T> and Expression<Func<T>>, the compiler handles it for you. But if you introduce your own function call into a LINQ statement, you do need to get it right.

Community
  • 1
  • 1
Matt Warren
  • 10,279
  • 7
  • 48
  • 63
  • As a test, I changed the code back to IEnumerable, and things appear to still be working. In the Raven console, I see pageSize=50, so I would guess that means Take() is making it to the Raven server. I'm now wondering if the difference is between session.Query and session.Advanced.LuceneQuery. I'm trying to make this not work again to increase my understanding, but it's still working. Grrr... – Bob Horn Jan 29 '12 at 19:09
  • The best clue is to use Intellisense and mouse-overs to get Visual Studio to tell you what version of a function is being called. Do it one the .Where(..) and .OrderBy(..) functions. Also looking at the Raven console is a good idea (as you're already doing) – Matt Warren Jan 29 '12 at 19:26
  • Yeah, even when using IEnumerable, OrderByDescending() is IRavenQueryable, and Take() is IQueryable. So I guess it wasn't just IEnumerable vs IQueryable all along. Looks like I only needed IQueryable if using LuceneQuery. – Bob Horn Jan 29 '12 at 20:15
  • I'm wondering now if it's even possible to do this with a LuceneQuery. When trying a LuceneQuery, OrderByDescending() is an extension method on IEnumerable, and Take() won't work because the compiler can't convert from IEnumerable to IQueryable. – Bob Horn Jan 29 '12 at 21:08
1

RavenDB uses eventual consistency by default, so indexes can be stale unless you explicitly specify otherwise.

Add the line below (or one of it's variants) to your query to this:

  .Customize(x => x.WaitForNonStaleResultsAsOfNow())
Matt Warren
  • 10,279
  • 7
  • 48
  • 63
  • Thanks, Matt. I should be able to give that a try at some point today. I'll let you know if it works. – Bob Horn Jan 20 '12 at 12:35
  • Ahh Matt - i thought QueryAndCacheEtags was some secret advanced RavenDb method. Not a custom one :P (Cause i was thinking of the same answer as you, did) :P – Pure.Krome Jan 20 '12 at 13:59
  • Matt, does this work for a LuceneQuery? I can't get Customize() to show up in intellisense. – Bob Horn Jan 20 '12 at 15:43
  • Also, this problem still exists the next day, with a new instance of the UI running. I would think that anything that *was* stale would have been fresh by now. Still investigating... – Bob Horn Jan 20 '12 at 15:44
  • Okay, with a Lucene query, I just had to do this: .WaitForNonStaleResultsAsOfNow() (no Customize) But it's still not showing the most recent data. For some reason it stops as of 1/19/2012 6:33:54 PM. – Bob Horn Jan 20 '12 at 15:53