Make elasticsearch only return certain fields?

Question

I'm using elasticsearch to index my documents.

Is it possible to instruct it to only return particular fields instead of the entire json document it has stored?

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html#_source, note thay you can also exclude only some fields — Christophe Roussy, May 25 '16 at 13:30

score 809 · Accepted Answer · edited Nov 02 '20 at 01:01

809

Yep, Use a better option source filter. If you're searching with JSON it'll look something like this:

{
    "_source": ["user", "message", ...],
    "query": ...,
    "size": ...
}

In ES 2.4 and earlier, you could also use the fields option to the search API:

{
    "fields": ["user", "message", ...],
    "query": ...,
    "size": ...
}

This is deprecated in ES 5+. And source filters are more powerful anyway!

edited Nov 02 '20 at 01:01

answered Mar 07 '12 at 16:39

kevingessner

18,559
5
43
63

13

make sure to define them as "stored": true in mapping. Otherwise ES will still load the _source document and load fields from there. May impact performance if returned data is relatively small to the size of a whole document. – Zaar Hai Aug 18 '13 at 10:19
7

you meant "store": true – sscarduzio Feb 17 '14 at 16:50
are these made in the conf file or where exactly? – vbNewbie Feb 26 '14 at 17:26
@vbNewbie: Where ever you are defining mapping. If you aren't defining mapping explicitly and relying on ES to generate it, then you'll have to define mapping for fields you want ES to store. You can define mapping just for fields where you want special behavior (e.g. "store":true, "index":"not_analyzed") or all fields. Look into mapping docs for more details. – Sangharsh Mar 08 '14 at 08:13
Moreover, fields are indexed by default but not stored separately, only whole document is stored as "_source". How to put your mapping in ES, I think would depend on your application / how you are indexing documents. – Sangharsh Mar 08 '14 at 08:16
Is it possible to use "fields": ["user"] and make the result appear under "_source" property. Now I get the result under hits: [ { fields: { user: "someone" } } ] – Marcus Mar 21 '14 at 08:01
4

fields is no longer supported on newer versions. use stored_fields instead :) – Sachin Sharma Nov 01 '17 at 09:53
```{ "fields": ["user", "message", ...], "query": ..., "size": ... }worked for me – Aditya Goel Nov 08 '21 at 17:31
1

The `fields` option seems to not be deprecated in Elasticsearch 8.3 in 2022. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html – Noumenon Jul 22 '22 at 00:08
How do you use_source for specific document fields and also return aggregations? – Mathijs Jan 11 '23 at 10:32

score 102 · Answer 2 · edited Aug 05 '19 at 12:24

I found the docs for the get api to be helpful - especially the two sections, Source filtering and Fields: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/docs-get.html#get-source-filtering

They state about source filtering:

If you only need one or two fields from the complete _source, you can use the _source_include & _source_exclude parameters to include or filter out that parts you need. This can be especially helpful with large documents where partial retrieval can save on network overhead

Which fitted my use case perfectly. I ended up simply filtering the source like so (using the shorthand):

{
    "_source": ["field_x", ..., "field_y"],
    "query": {      
        ...
    }
}

FYI, they state in the docs about the fields parameter:

The get operation allows specifying a set of stored fields that will be returned by passing the fields parameter.

It seems to cater for fields that have been specifically stored, where it places each field in an array. If the specified fields haven't been stored it will fetch each one from the _source, which could result in 'slower' retrievals. I also had trouble trying to get it to return fields of type object.

So in summary, you have two options, either though source filtering or [stored] fields.

Did the trick for me. I had a problem with returning geo_point using "fields", but "_source" works just fine, thanks ! — Yonnaled, Aug 11 '16 at 08:28
How do you use_source for specific document fields and also return aggregations? — Mathijs, Jan 11 '23 at 10:32

score 32 · Answer 3 · edited Nov 25 '20 at 09:31

32

For the ES versions 5.X and above you can a ES query something like this:

    GET /.../...
    {
      "_source": {
        "includes": [ "FIELD1", "FIELD2", "FIELD3" ... " ]
      },
      .
      .
      .
      .
    }

edited Nov 25 '20 at 09:31

Lee Goddard

10,680
4
46
63

answered Apr 14 '17 at 15:40

Pinkesh Sharma

2,428
20
16

score 24 · Answer 4 · edited Jul 05 '22 at 06:50

24

here you can specify whichever field you want in your output and also which you don't:

  POST index_name/_search
    {
        "_source": {
            "includes": [ "field_name", "field_name" ],
            "excludes": [ "field_name" ]
        },
        "query" : {
            "match" : { "field_name" : "value" }
        }
    }

edited Jul 05 '22 at 06:50

keser

2,472
1
12
38

answered Sep 18 '17 at 10:53

Gaurav

597
6
15

score 15 · Answer 5 · answered Mar 29 '17 at 14:57

In Elasticsearch 5.x the above mentioned approach is deprecated. You can use the _source approach, but but in certain situations it can make sense to store a field. For instance, if you have a document with a title, a date, and a very large content field, you may want to retrieve just the title and the date without having to extract those fields from a large _source field:

In this case, you'd use:

{  
   "size": $INT_NUM_OF_DOCS_TO_RETURN,
   "stored_fields":[  
      "doc.headline",
      "doc.text",
      "doc.timestamp_utc"
   ],
   "query":{  
      "bool":{  
         "must":{  
            "term":{  
               "doc.topic":"news_on_things"
            }
         },
         "filter":{  
            "range":{  
               "doc.timestamp_utc":{  
                  "gte":1451606400000,
                  "lt":1483228800000,
                  "format":"epoch_millis"
               }
            }
         }
      }
   },
   "aggs":{  

   }
}

See the documentation on how to index stored fields. Always happy for an Upvote!

score 12 · Answer 6 · edited Dec 23 '20 at 06:48

Here is another solution, now using a match expression

Source filtering allows to control how the _source field is returned with every hit.

Tested with Elastiscsearch version 5.5

The keyword includes defines the specifics fields.

GET /my_indice/my_indice_type/_search
{
  "_source": {
    "includes": [
      "my_especific_field"
    ]
  },
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "_id": "%my_id_here_without_percent%"
          }
        }
      ]
    }
  }
}

The Demz · Answer 7 · 2021-04-26T15:17:36.440

9

response_filtering

All REST APIs accept a filter_path parameter that can be used to reduce the response returned by elasticsearch. This parameter takes a comma separated list of filters expressed with the dot notation.

https://stackoverflow.com/a/35647027/844700

edited Apr 26 '21 at 15:17

answered Feb 26 '16 at 08:50

The Demz

7,066
5
39
43

score 6 · Answer 8 · answered Aug 10 '17 at 08:00

A REST API GET request could be made with '_source' parameter.

Example Request

http://localhost:9200/opt_pr/_search?q=SYMBOL:ITC AND OPTION_TYPE=CE AND TRADE_DATE=2017-02-10 AND EXPIRY_DATE=2017-02-23&_source=STRIKE_PRICE

Response

{
"took": 59,
"timed_out": false,
"_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
},
"hits": {
    "total": 104,
    "max_score": 7.3908954,
    "hits": [
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLc",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 160
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLh",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 185
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLi",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 190
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLm",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 210
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLp",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 225
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLr",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 235
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLw",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 260
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uL5",
            "_score": 7.3908954,
            "_source": {
                "STRIKE_PRICE": 305
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLd",
            "_score": 7.381078,
            "_source": {
                "STRIKE_PRICE": 165
            }
        },
        {
            "_index": "opt_pr",
            "_type": "opt_pr_r",
            "_id": "AV3K4QTgNHl15Mv30uLy",
            "_score": 7.381078,
            "_source": {
                "STRIKE_PRICE": 270
            }
        }
    ]
}

}

The result shown above indicates a total hit of 104. Yet only 11 are shown up. How come? — wh81752, Sep 12 '22 at 12:59
I used this approach a lot, and always struggle to find any doc for it, thanks! — Felix, Jun 14 '23 at 14:28

score 6 · Answer 9 · answered Sep 28 '18 at 05:31

Yes by using source filter you can accomplish this, here is the doc source-filtering

Example Request

POST index_name/_search
 {
   "_source":["field1","filed2".....] 
 }

Output will be

{
  "took": 57,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "index_name",
        "_type": "index1",
        "_id": "1",
        "_score": 1,
        "_source": {
          "field1": "a",
          "field2": "b"
        },
        {
          "field1": "c",
          "field2": "d"
        },....
      }
    ]
  }
}

Miguel Barrios · Answer 10 · 2022-04-19T10:13:35.657

There are several methods that can be useful to achieve field-specific results. One can be through the source method. And another method that can also be useful to receive cleaner and more summarized answers according to our interests is filter_path:

Document Json in index "index1":

"hits" : [
  {
    "_index" : "index1",
    "_type" : "_doc",
    "_id" : "1",
    "_score" : 1,
    "_source" : {
      "year" : 2020,
      "created_at" : "2020-01-29",
      "url" : "www.github.com/mbarr0987",
      "name":"github"
    }
  }

Query:

GET index1/_search?filter_path=hits.hits._source.url
{
  "query": { 
        {"term": {"name":"github" }
    }
  }
}

Output:

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "url" : "www.github.com/mbarr0987"
            }
          }
      ]
   }
}

score 4 · Answer 11 · answered Mar 30 '17 at 13:19

4

In java you can use setFetchSource like this :

client.prepareSearch(index).setTypes(type)
            .setFetchSource(new String[] { "field1", "field2" }, null)

answered Mar 30 '17 at 13:19

user1693371

124
1
4

score 3 · Answer 12 · edited May 13 '20 at 00:09

For example, you have a doc with three fields:

PUT movie/_doc/1
{
  "name":"The Lion King",
  "language":"English",
  "score":"9.3"
}

If you want to return name and score you can use the following command:

GET movie/_doc/1?_source_includes=name,score

If you want to get some fields which match a pattern:

GET movie/_doc/1?_source_includes=*re

Maybe exclude some fields:

GET movie/_doc/1?_source_excludes=score

score 1 · Answer 13 · answered Aug 06 '20 at 03:39

if you know sql, please write a query to get the code's value,for example sql query equivalent and elasticsearch query

POST /_sql/translate
{
  
  "query": "select name,surname from users"
}

result is ,be carefull look at the includes key

{
  "size" : 1000,
  "_source" : {
    "includes" : [
      "name",
      "surname"
    ],
    "excludes" : [ ]
  },
  "sort" : [
    {
      "_doc" : {
        "order" : "asc"
      }
    }
  ]
}

score 0 · Answer 14 · answered Mar 07 '23 at 16:29

0

GET /_cat/indices?format=json&h=<comma_separated_field_names>

answered Mar 07 '23 at 16:29

Aman

372
4
13

score 0 · Answer 15 · answered Mar 24 '23 at 13:57

0

In case, if anyone is looking for answer in python -

query = MatchAll()

queryset = Search(using=self.client, index=self.index)
queryset = queryset.source(includes=included_fields, excludes=excluded_fields)
queryset = queryset.query(query)

answered Mar 24 '23 at 13:57

Sachin

103
7

Make elasticsearch only return certain fields?

15 Answers15

Linked