Add documents to elasticsearch if it does not exists

Question

I transferred some data from a log generated every day to elasticsearch using logstash, and my logstash output section looks like :

i keep the same id (id_ot) in both my log file and elasticsearch, but what i would like to do is : if the new coming id ( id_ot) already exists in elasticseach, so i will not insert it. How can i do that in logstash ?

Any help would be really appriciated !

score 2 · Accepted Answer · answered Aug 27 '19 at 13:54

2

You simply need to add the action => create parameter and if a document already exists with that ID, it is not indexed

output {
  elasticsearch {
    ...
    action => "create"
  }
}

answered Aug 27 '19 at 13:54

Val

207,596
13
358
360

Thank you Val. that's all what i need. Regards – Mohamed Aug 28 '19 at 09:20
Val, if i may ask you another question: I implemented ILM for managing the data which looks like: – Mohamed Aug 28 '19 at 09:52
1

Mohamed, let's close this question and create a new one, so we don't cram too much unrelated topics into this thread, thanks – Val Aug 28 '19 at 09:53

score -1 · Answer 2 · edited Aug 27 '19 at 21:09

-1

What you are asking for is an upsert , create the document if dosent exist or update an existing one . elastic supports this via doc_as_upsert option

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#doc_as_upsert

on the right hand side in the link you can choose the elastic version that matches your version.

edited Aug 27 '19 at 21:09

Polynomial Proton

5,020
20
37

answered Aug 27 '19 at 16:08

Shachaf.Gortler

5,655
14
43
71

5

`doc_as_upsert` creates document if its doesnt exist. But if its exists - it will update the document. OP doesnt want to update existing but only create. So Val's answer is the correct. `create` action doesnt update for existing ones. – Polynomial Proton Aug 27 '19 at 21:11

Add documents to elasticsearch if it does not exists

2 Answers2