Elastic search : query to get all elements

Question

I can't get all the items, the maximum reached is size:10000. thanks

Error: [query_phase_execution_exception] Result window is too large, from + size must be less than or equal to: [10000] but was [90000]. See the scroll API for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.

Any idea how can I solve it?

GetTweets: function (callback) {
        client.search({
            index: 'twitter',
            type: 'tweet',
            size:10000,
            body: {
                query: {
                    "query": {
                        "match_all": {}
                    }
                }
            }
        }, function (err, resp, status) {
            callback(err,resp);
        });
    },

wonder · Answer 1 · 2018-12-01T19:33:09.710

search_after can be used to apply pagination.Efficient than Scroll Api

GET twitter/_search
{
    "size": 10,
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    },
    "search_after": [1463538857, "654323"],
    "sort": [
        {"date": "asc"},
        {"tie_breaker_id": "asc"}
    ]
 }

ES docs: It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher

A l w a y s S u n n y · Accepted Answer · 2018-12-01T15:26:28.927

It is the default feature of Elasticsearch not to get data at once after 10000 window ie. size:10000 or upper. See here at scroll api, because of that restriction you're getting below error.

Result window is too large, from + size must be less than or equal to: [10000]

Try Scroll API like,

curl -XGET 'localhost:9200/twitter/tweet/_search?scroll=1m' -d '
{
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}
'

The result from the above request includes a _scroll_id, which should be passed to the scroll API in order to retrieve the next batch of results.

curl -XGET  'localhost:9200/_search/scroll'  -d'
{
    "scroll" : "1m", 
    "scroll_id" : "c2Nhbjs2OzM0NDg1ODpzRlBLc0FXNlNyNm5JWUc1" 
}
'

N.B I've used both the python and php version of elasticsearch client api. Scroll API is really awesome and very flexible to get data-sets using it.

Elastic search : query to get all elements

2 Answers2