Mutliple config files causing duplicate message

Question

I have a Logstash machine running in AWS. In Logstash I have 3 config files each having 1 input defined on them. These inputs are reading logs from following sources

From s3
From http input
From filebeat

The problem is that I am getting duplicate messages in Kibana. So for 1 message generated by Filebeat I am seeing 3 messages in Kibana. I tried to remove 1 config file and the count got reduced to 2. So I am pretty sure that this is due to these config files.

What is confusing me is that why this is happening. I have separate input defined on all 3 config files, still getting duplicate messages. These are the input section of all 3 config files.

s3 input

input {
  s3 {
     bucket => "elb-logs"
     region => "us-east-1"
     prefix => "demo/AWSLogs/792177735214/"
     type   => "elb-logs"
     delete => true
  }

}

Http input

input {
  http {
     type   => "frontend-logs"
     codec  => "json"
  }
}

Filebeat

input {
  beats {
    port => "5043"
  }
}

For all 3 config files there is common output section i.e.

output {
    elasticsearch { hosts => [ "10.0.0.1:9200" ] }
}

@OllyCruickshank I have edited my question with the output configured by me. — shivams, May 04 '17 at 11:56
logstash will concatenate the config files together and see three output sections. As a result your message will be output three times. i would create a separate, single output config file and remove the output section from your 3 input config files. — Olly Cruickshank, May 04 '17 at 11:58
@OllyCruickshank Thanks, it worked. Can you please give this as an answer so that i can accept it. — shivams, May 04 '17 at 12:35

score 3 · Accepted Answer · answered May 04 '17 at 13:41

Logstash will concatenate the three config files together (s3 input, Http input, Filebeat) and see three output sections.

The three output sections are not related to the specific inputs - instead Logstash will send an input from any one of the three sources to all of the configured outputs. As a result your message will be output three times to the same destination.

I would create a separate, single output config file and remove the output section from your 3 input config files.

Mutliple config files causing duplicate message

s3 input

Http input

Filebeat

1 Answers1

Linked