elasticsearch: define field's order in returned doc

Question

i'm doing sending queries to elasticsearch and it responde with an unknown order of fields in its documents. how can i fix the order that elsasticsearch is returning fields inside documents? i mean, i'm sending this query:

{
"index": "my_index",
"_source":{
          "includes" : ["field1","field2","field3","field14"]
  },
"size": X,
"body": {
    "query": { 
        // stuff    
       }
    }
}

and when it responds, it gives me something not in the good order. i ultimatly want to convert this to csv, and want to fix csv headers. is there something to do so i can get something like doc1 :{"field1","field2","field3","field14"} doc2 :{"field1","field2","field3","field14"} ... in the same order as my "_source" ?

thank's for your help.

Answer 1

A document in Elasticsearch is a JSON hash/map and by definition maps are unordered.

One solution around this would be to use Logstash in order to extract docs from ES using an elasticsearch input and output them in CSV using a csv output. That way you can guarantee that the fields in the CSV file will have the exact same order as specified. Another benefit is that you don't have to write your own boilerplate code to extract from ES and sink to CSV, Logstash does it all for you for free.

The Logstash configuration would look something like this:

input {
  elasticsearch {
    hosts => "localhost"
    query => '{ "query": { "match_all": {} } }'
    size => 100
    index => "my_index"
  }
}
filter {}
output {
    csv {
        fields => ["field1","field2","field3","field14"]
        path => "/path/to/file.csv"
    }
}