Counting number of documents using Elasticsearch

Question

If one wants to count the number of documents in an index (of Elasticsearch) then there are (at least?) two possibilities:

Direct count

POST my_index/_count

should return the number of documents in my_index.
Using search

Here one can use the count as the search_type or some other type. In either of the cases the total count can be extracted from the field ['hits']['total']

My questions are:

what is the difference between the different approaches? Which one should I prefer?
I raise this question because I'm experiencing different results depending on the chosen method. I'm now in the process of debugging the issue, and this question popped up.

Getting the count is a GET request: `{"count":27053653,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0}}` — MortenB, Nov 15 '18 at 14:16

score 67 · Accepted Answer · edited Sep 10 '14 at 09:59

67

Probably _count is a bit faster since it doesn't have to execute a full query with ranking and result fetching and can simply return the size.

It would be interesting to know a bit more about how you manage to get different results though. For that I need more information like what exact queries you are sending and if any indexing is going on on the index.

But suppose that you do the following

index some documents
refresh the index

_search and _count (with a match all query) should return the same total. If not, that'd be very weird.

edited Sep 10 '14 at 09:59

Dror

12,174
21
90
160

answered Sep 09 '14 at 11:37

Jilles van Gurp

7,927
4
38
46

Unfortunately I won't be able to share the data. Furthermore, since the problem was not 100% reproducible, it would be hard to come up with a minimal example. That's why I asked it as a general question. – Dror Sep 10 '14 at 09:37
won't need the data of course, just anonymize it. But the actual request + response would be useful. Without that it is going to be pretty hard to figure out what you are doing wrong. – Jilles van Gurp Sep 10 '14 at 13:37
1

apparently _count api is being deprecated in es 2.0 for reasons of being redundant given that you can search with size=0 – Jilles van Gurp Aug 03 '15 at 14:11
@JillesvanGurp are you sure _count is deprecated in version 2? ES documentation is pretty good at including deprecation notices, and there is none for _count. – huwiler Sep 16 '15 at 15:37
_count definitely been removed in v5, I believe. Here is the ticket where the removal is discussed: https://github.com/elastic/elasticsearch/issues/13928. Searchtype=_count is gone in 5.1 as well. – Jilles van Gurp Dec 14 '16 at 14:26
1

The `_count` is there, even in version 7. Not sure about the clients, but in ES itself Count API endpoint has not been removed. See: https://www.elastic.co/guide/en/elasticsearch/reference/7.1/search-count.html – wst Jun 10 '19 at 20:28
It seems you are right. They've been debating this for ages: https://github.com/elastic/elasticsearch/issues/13928 – Jilles van Gurp Jun 11 '19 at 12:51

score 47 · Answer 2 · answered Mar 25 '20 at 20:59

47

If _search must be used instead of _count, and you're on Elasticsearch 7.0+, setting size: 0 and track_total_hits: true will provide the same info as _count

GET my-index/_search
{
  "query": { "term": { "field": { "value": "xyz" } } },
  "size": 0,
  "track_total_hits": true
}


{
  "took" : 612,
  "timed_out" : false,
  "_shards" : {
    "total" : 629,
    "successful" : 629,
    "skipped" : 524,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 29349466,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

See Elasticsearch 7.0 Breaking changes

answered Mar 25 '20 at 20:59

theY4Kman

5,572
2
32
38

If there are 10,000+ documents that match and if I only want to retrieve the first 10,000 as set by `index.max_result_window` but want to get the actual count, would it be faster to set `track_total_hits: true` or if the hits count > 10,000 then issue a `_count` query to get the actual count? – Dula Aug 06 '21 at 21:01
hmm well I'm testing this on Elastic 8.5 but count returns all items in index and search if we limit the results by query, set size =0 , track_total_hits: true is returning count of what the query would normally return (lets say term query which will limit results a bit). In my case count = 160 k and search with mentioned params = 130 k. So i don't see how its the same thing. I guess it would be if query would be a matchAll query. – cah1r Jan 02 '23 at 13:45

score 34 · Answer 3 · answered Oct 09 '19 at 20:56

curl http://localhost:9200/_cat/indices?v provides you the count and other information in a tabular format

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2019.10.09-000001         IS7HBUgRRzO7Rn1puBFUIQ   1   1          0            0       283b           283b
green  open   .kibana_task_manager_1             e4zZcF9wSQGFHB_lzTszrg   1   0          2            0     12.5kb         12.5kb
yellow open   metricbeat-7.4.0-2019.10.09-000001 h_CWzZHcRsakxgyC36-HTg   1   1       6118            0      2.2mb          2.2mb
green  open   .apm-agent-configuration           J6wkUr2CQAC5kF8-eX30jw   1   0          0            0       283b           283b
green  open   .kibana_2                          W2ZETPygS8a83-Xcd6t44Q   1   0       1836           23      1.1mb          1.1mb
green  open   .kibana_1                          IrBlKqO0Swa6_HnVRYEwkQ   1   0          8            0    208.8kb        208.8kb
yellow open   filebeat-7.4.0-2019.10.09-000001   xSd2JdwVR1C9Ahz2SQV9NA   1   1          0            0       283b           283b
green  open   .tasks                             0ZzzrOq0RguMhyIbYH_JKw   1   0          1            0      6.3kb          6.3kb

Note also that `format=json` or `format=yaml` can be provided as a query param to return CAT endpoint results in machine-readable format. Additionally, the returned columns may be filtered with `h=index,docs.count`, so a succinct way to retrieve doc counts for many indices might be: `curl http://localhost:9200/_cat/indices?h=index,docs.count&format=json` — theY4Kman, Jul 13 '21 at 20:09

Adeel Siddiqui · Answer 4 · 2020-01-20T11:06:33.843

24

Old question, chipping in because on ElasticSearch version > 7.0 :

_search: returns the documents with the hit count for the search query, less than or equal to the result window size, which is typically 10,000. e.g.:

{"took":3,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score": 0.34027478,"hits":[...]}}
_count: returns the total number of hits for the search query irrespective of the result window size. no documents returned, e.g.:

{"count":5703899,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

So, _search might return the total hits as 10,000 if that is your configured result window size, while _count would return the actual count for the same query.

edited Jan 20 '20 at 11:06

answered Jan 20 '20 at 10:52

Adeel Siddiqui

676
7
16

2

Thank you. This should be the accepted answer as _search will never return a count above 10,000. – Julien Le Coupanec Jun 06 '20 at 17:39
1

disagree, see https://stackoverflow.com/a/60857312/373254 above – sly7_7 Mar 30 '21 at 13:54
Thank you! Is there a way to implement the `Count` api in NEST (.net client)? I know if you do `GET /INDEX*/_count` that will return the count but how would you do that for nest? – MarkCo Jul 16 '21 at 14:38
@MarkCo No idea, haven't used the .net client :) – Adeel Siddiqui Jul 18 '21 at 15:57

score 2 · Answer 5 · answered Sep 09 '14 at 13:29

2

The two queries provide the same result but: - count consumes less resources/bandwidth because doesn't require to fetch documents, scoring and other internal optimizations. Set the search size to 0, could be very similar.

If you want count all the record in an index, you can also execute an aggregation terms on "_type" field.

The results should be the same. Before comparing the results, be sure to execute an index refresh.

answered Sep 09 '14 at 13:29

aparo

84
2

The terms aggregation has the pit hole of the accuracy. You have to set a large size, and it is always bounded from above by `MAX_INT`... – Dror Sep 09 '14 at 13:43

score -1 · Answer 6 · answered Sep 21 '22 at 11:32

If you want to check index by index, you can use the following query

GET _all/_search
{
  "size": 0, 
  "aggs": {
    "NAME": {
      "terms": {
        "field": "_index",
        "size": 100000
      }
    }
  }
}

The result will be the following screenshot.

Counting number of documents using Elasticsearch

6 Answers6

Linked