Add a header line before each json document

Question

I have a json file with 1000 json object. is there any way to add a header line before each json document ? Is there any easiest way ?

Example : I have 1000 object like this

{"id":58,"first_name":"Louis","last_name":"Jordan","email":"ljordan1l@nature.com","gender":"Male","Latitude":"-15.93444","Longitude":"-50.14028"}

i want to add index header like below for every json object so that i can use in Elasticsearch Bulk api

{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "unique_id" } }
{"id":58,"first_name":"Louis","last_name":"Jordan","email":"ljordan1l@nature.com","gender":"Male","Latitude":"-15.93444","Longitude":"-50.14028"}

Do you want to specify your own IDs, or are you willing to let ES generate them automatically for you? — Val, Nov 14 '16 at 10:10

Val · Accepted Answer · 2016-11-14T10:32:06.130

If you are willing to leverage Logstash, you don't need to modify your file and can simply read it line by line and stream it to ES using the elasticsearch output which leverages the Bulk API.

Store the following Logstash configuration in a file named es.conf (make sure the file path and ES hosts match your settings):

input {
  file {
    path => "/path/to/your/json"
    sincedb_path => "/dev/null"
    start_position => "beginning"
    codec => "json"
  }
}
filter {
  mutate {
    remove_fields => ["@version", "@timestamp"]
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "test"
    document_type => "type1"
    document_id => "%{id}"
  }
}

Then, you need to install logstash and you'll be able to run the following command in order to load your JSON files to your ES server:

bin/logstash -f es.conf

How can i run this snippet ? – Md Hafezur Rahman Nov 14 '16 at 10:28 — Md Hafezur Rahman, Nov 14 '16 at 10:28

score 0 · Answer 2 · edited May 23 '17 at 12:01

0

I found the best way to Add a header line before each json document. https://stackoverflow.com/a/30899000/5029432

edited May 23 '17 at 12:01

Community

1
1

answered Nov 14 '16 at 11:31

Md Hafezur Rahman

376
1
6
15

Cool, note, however, that one disadvantage of that solution is that it doesn't allow to specify a document ID. – Val Nov 14 '16 at 12:05
@Val, when you import file by BULK API, id will be auto generate. – Md Hafezur Rahman Nov 15 '16 at 04:50
I know that, I was just mentioning this for those people stumbling upon this question with the need to specify their own id. – Val Nov 15 '16 at 04:52

Add a header line before each json document

2 Answers2

Linked