How to get all documents under an elasticsearch index with python client ?

Question

I'm trying to get all index document using python client but the result show me only the first document This is my python code :

res = es.search(index="92c603b3-8173-4d7a-9aca-f8c115ff5a18", doc_type="doc", body = {
'size' : 10000,
'query': {
    'match_all' : {}
}
})
print("%d documents found" % res['hits']['total'])
data = [doc for doc in res['hits']['hits']]
for doc in data:
    print(doc)
    return "%s %s %s" % (doc['_id'], doc['_source']['0'], doc['_source']['5'])

seems like there is only 1 doc for `doc_type=doc`. can you recheck. — harshil9968, May 07 '18 at 08:54
This request must show me 3 documents the first print show me " 3 documents found" — J.Ghassen, May 07 '18 at 09:06

score 8 · Answer 1 · edited Nov 15 '20 at 17:43

8

try "_doc" instead of "doc"

res = es.search(index="92c603b3-8173-4d7a-9aca-f8c115ff5a18", doc_type="_doc", body = {
'size' : 100,
'query': {
    'match_all' : {}
}
})

edited Nov 15 '20 at 17:43

Suraj Rao

29,388
11
94
103

answered Nov 15 '20 at 17:40

Habib Mezghani

113
1
4

Lupanoide · Answer 2 · 2018-05-07T09:10:36.797

4

Elasticsearch by default retrieve only 10 documents. You could change this behaviour - doc here . The best practice for pagination are search after query and scroll query. It depends from your needs. Please read this answer Elastic search not giving data with big number for page size

To show all the results:

for doc in res['hits']['hits']:
    print doc['_id'], doc['_source']

edited May 07 '18 at 09:10

answered May 07 '18 at 08:55

Lupanoide

3,132
20
36

1

I have only 3 documents to get from this index – J.Ghassen May 07 '18 at 09:07
The problem is with the return statement of my function so how can I return the result correctly ? – J.Ghassen May 07 '18 at 09:32
just return data, which is defined as data = [doc for doc in res['hits']['hits']] – Lupanoide May 07 '18 at 09:33
I want to return this result "%s %s %s" % (doc['_id'], doc['_source']['0'], doc['_source']['5']) – J.Ghassen May 07 '18 at 09:39

score 0 · Answer 3 · answered Mar 03 '21 at 16:10

0

You can try the following query. It will return all the documents.

result = es.search(index="index_name", body={"query":{"match_all":{}}})

answered Mar 03 '21 at 16:10

Tushar Nitave

519
4
13

score 0 · Answer 4 · answered Jun 19 '21 at 10:15

0

You can also use elasticsearch_dsl and its Search API which allows you to iterate over all your documents via the scan method.

import elasticsearch
from elasticsearch_dsl import Search

client = elasticsearch.Elasticsearch()
search = Search(using=client, index="92c603b3-8173-4d7a-9aca-f8c115ff5a18")

for hit in search.scan():
    print(hit)

answered Jun 19 '21 at 10:15

kluu

2,848
3
15
35

search.scan() can go through all docs but it is very slow. Is there a way to improve it? – H.C.Chen Nov 04 '21 at 07:12
Now I found the doc https://elasticsearch-dsl.readthedocs.io/en/latest/search_dsl.html#pagination seems like that's the best it can do. – H.C.Chen Nov 04 '21 at 08:00
It is unfortunate that this is as fast as it goes. I would be much interested if there was a way to speed things up? – j7skov Feb 08 '22 at 21:20

score 0 · Answer 5 · answered Oct 01 '22 at 03:31

0

I dont see mentioned that the index must be refreshed if you just added data. Use this:

es.indices.refresh(index="index_name")

answered Oct 01 '22 at 03:31

juniper25

63
5

How to get all documents under an elasticsearch index with python client ?

5 Answers5