Filebeat is processing all the logs instead of the specified application logs

Question

I have an app server, where I have configured filebeat(through Chef) to extract the logs and publish it to logstash(a separate ELK server), and subsequently to ES and Kibana.

I have configured filebeat to process logs only from /opt/app_logs/*.log, but it seems it is reading logs from other locations too, because in the /etc/filebeat configuration directory, I have filebeat.full.yml and other yml files generated automatically, and they seem to have all those other file locations, thus due to such a huge amount of logs, logstash service is getting out of memory within minutes with logstash.log. How can I not autogenerate the other yml files? I tried to remove this file and also tried to comment out all the /var/log paths from the prospectors, but then filebeat itself is not starting.

filebeat.yml file:

filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat"
output:
  logstash:
    hosts:
    - elk_host:5044
    index: logstash-filebeat
shipper:
  name: serverA
  tags:
  - A
logging:
  to_files: 'true'
  files:
    path: "/var/log/filebeat"
    name: filebeat_log
    rotateeverybytes: '10485760'
  level: info
prospectors:
- paths:
  - "/opt/app_logs/*.log"
  encoding: plain
  input_type: log
  ignore_older: 24h

What version of Filebeat are you using? Your config is incorrect for all versions, but knowing the version you are targeting can help in getting you the right answer. — A J, Jan 13 '17 at 17:01
@A J Sorry for replying so late. The version of filebeat is 1.2.3 (amd64) — tanmayghosh2507, Jan 16 '17 at 09:47

score 2 · Answer 1 · edited May 23 '17 at 11:45

The main problem with your configuration is that for Filebeat 1.2.3 you have the prospectors list defined twice and second one is not in the correct location.

The second problem is that you have defined the config_dir as /etc/filebeat. config_dir is used to specify an additional directory where to look for config files. It should never be set to /etc/filebeat because this is where the main config file should be located. See https://stackoverflow.com/a/39987501/503798 for usage information.

A third problem is that you have used string types in to_files and rotateeverybytes. They should be boolean and integer types respectively.

Here's how the config should look for Filebeat 1.x.

filebeat:
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
  prospectors:
  - paths:
    - "/opt/app_logs/*.log"
    encoding: plain
    input_type: log
    ignore_older: 24h
output:
  logstash:
    hosts:
    - elk_host:5044
    index: logstash-filebeat
shipper:
  name: serverA
  tags:
  - A
logging:
  to_files: true
  files:
    path: "/var/log/filebeat"
    name: filebeat_log
    rotateeverybytes: 10485760
  level: info

I highly recommend that you upgrade to Filebeat 5.x because it has better configuration validation using filebeat -configtest.

Thanks, This helped! – tanmayghosh2507 Jan 19 '17 at 05:46 — tanmayghosh2507, Jan 19 '17 at 05:46

Filebeat is processing all the logs instead of the specified application logs

1 Answers1