0

I have implemented pagination with Lucene using the searchAfter method provided by IndexSearcher. In every call, I pass the last ScoreDoc returned in the previous page.

The problem is that sometimes, the index gets updated between page and page and occasionally I am getting this exception:

java.lang.IllegalArgumentException: after.doc exceeds the number of 
documents in the reader: after.doc=337 limit=337
    at 
org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:434)

I understand that Lucene changes the docs ids every now and then (segment merges, etc.) and I guess that is why that exception is happening as searchAfter relies on those docs ids.

How could I improve this pagination mechanism to avoid this exception? Is there any better way to implement pagination with Lucene?

Ivan
  • 1,477
  • 3
  • 18
  • 36
  • how many times you see this problem occurs? like how many times you get index updates while searching – dom Nov 13 '17 at 13:57
  • @dom no very often. I have implemented some tests that update the index (delete and/or update documents) during searching and many time it doesn't fail. It depends on when Lucene changes those docs internally. – Ivan Nov 13 '17 at 14:10

1 Answers1

1

I'm not familiar with this error but i would suggest to implement a way you can use NRT (Near real time) searches with a SearcherManager (JavaDoc)

it's a big effort but at the end you receive a better update handling for indices especially if they update more frequently.

See: Lucene NRT beginner tutorial or related stackoverflow answer

dom
  • 732
  • 7
  • 19
  • Actually I am using NRT and SearcherManager. I acquire a SearcherManager when a new page needs to be fetched and then I call searchAfter(...). – Ivan Nov 13 '17 at 15:34
  • do you use maybeRefresh() after indexing? – dom Nov 13 '17 at 16:09