0

I am trying to figure out how to retrieve possibly a large amount of documents using NEST 2.3.2. Because there is a limit on the number of documents in a response (set by the server admin), I want to use Scroll to be safe.

I've read this but I didn't find a way to achieve that using NEST 2.3.2 library. This documentation is for 1.x, and it doesn't work for 2.x.

Specifically, when I try to set SearchType = Elasticsearch.Net.SearchType.Scan in the SearchRequest, it automatically sends request to /_search?search_type=scan and the server just returns "search_phase_execution_exception" because "Scroll must be provided when scanning...". But I don't know how to get a Scroll ID.

Does anyone know how to achieve this? I want to scroll over the qualified documents with possibly several requests until they are all retrieved. If anyone can provide some sample code that would be very helpful. Thank you very much.

LLS
  • 2,128
  • 2
  • 23
  • 36

1 Answers1

2

I tried something like this for my problem in which I need to get all the documents indexed and then do some processing. I used scroll. You can have a look at my question from here

var scanResults = client.Search<IndexName>(s => s
                .From(0)
                .Size(20) //any size you can give
                .MatchAll()
                .Source(so => so
                .Include(fi => fi.Field(fieldName))
                .SearchType(Elasticsearch.Net.SearchType.Scan)
                .Scroll("5m") //time for which the scrollId is saved. Can be anything.
            );

            var results = client.Scroll<IndexName>("10m", scanResults.ScrollId);

            while (results.Documents.Any())
            {
                //do whatever you want to do from the resultant documents

                results = client.Scroll<IndexName>("10m", results.ScrollId);
            }

More information about scroll

Hope this helps.

Community
  • 1
  • 1
ASN
  • 1,655
  • 4
  • 24
  • 52
  • 1
    Thank you for your answer. Well, I figured it out by looking at the plaintext version and guessing the parameters in the NEST API. The key is to set the `Scroll` property in the `SearchRequest` object and pass it to the `Search` method. If `Scroll` is set to a nonempty value, it generates an id. I still need to figure out the details but I think it should work. – LLS Jun 22 '16 at 07:06
  • May be you can give it a try and check if this helps you. – ASN Jun 22 '16 at 07:13