3

i have duplicate data in Logstash

how could i remove this duplication?

my input is:

input

input {
  file {
    path => "/var/log/flask/access*"
    type => "flask_access"
    max_open_files => 409599
  }
  stdin{}
}

filter

filter of files is :

filter {
  mutate { replace => { "type" => "flask_access" } }
  grok {
    match => { "message" => "%{FLASKACCESS}" }
  }
  mutate {
    add_field => {
      "temp" => "%{uniqueid} %{method}"
    }
  }
   if "Entering" in [api_status] {
     aggregate {
       task_id => "%{temp}"
       code => "map['blockedprocess'] = 2"
       map_action => "create"
     }
   }
   if "Entering" in [api_status] or "Leaving" in [api_status]{
     aggregate {
       task_id => "%{temp}"
       code => "map['blockedprocess'] -= 1"
       map_action => "update"
     }
   }
   if "End Task" in [api_status] {
     aggregate {
       task_id => "%{temp}"
       code => "event['blockedprocess'] = map['blockedprocess']"
       map_action => "update"
       end_of_task => true
       timeout => 120
     }
   }
 }

Take a look at the image, the same data log, at the same time, and I just sent one log request.

enter image description here

Mikail Land
  • 269
  • 3
  • 9

2 Answers2

1

i solve it

i create a unique id by ('document_id') in output section

document_id point to my temp and temp is my unique id in my project

my output changed to:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    document_id => "%{temp}"
#    sniffing => true
#    manage_template => false
#    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
#    document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}
Mikail Land
  • 269
  • 3
  • 9
  • 1
    Did you have multiple conf files? If so, the output section in are merged together creating multiple output sections. Mentioned here: https://stackoverflow.com/questions/43781955/mutliple-config-files-causing-duplicate-message – andho Jul 09 '17 at 17:21
0

Executing tests in my local lab, I've just found out that logstash is sensitive to the number of its config files that are kept in /etc/logstash/conf.d directory. If config files are more than 1, then we can see duplicates for the same record.

So, try to remove all backup configs from /etc/logstash/conf.d directory and perform logstash restart.

Andrey
  • 1