4

I have a simple Fluentd-Elasticsearch-Kibana setup which has a very strange behaviour. Fluentd seems to stop sending information to elastic search after 3 hours it is up.

I run everything in a simple docker-compose file

version: '2'
services:

fluentd:
  build: ./fluentd
  volumes:
    - ./fluentd/conf:/fluentd/etc
  links:
    - "elasticsearch"
  ports:
    - "24225:24225"
    - "24225:24225/udp"

elasticsearch:
  image: elasticsearch
  expose:
    - 9200
  ports:
    - "9200:9200"

kibana:
  image: kibana
  links:
    - "elasticsearch"
  ports:
    - "5601:5601"

The Fluentd is built following this dockerfile

# fluentd/Dockerfile
FROM fluent/fluentd:v0.12-debian
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri", "--version", "1.9.2"]

and it has the following conf file:

<source>
  @type forward
  port 24225
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

All running in the cloud, and, of course, in the same machine. From other machines/instances, I send my logs, and Fluentd does receive those logs without any issues. The problem is that after every 3 hours, the Fluentd suddenly stops forwarding those logs to my Elasticsearch. No error message, nothing. If I do restart the Fluentd container, everything works for the next 3 hours.

I looked for this kind of behaviour, but could not find anywhere an explanation or someone with a situation that would come close to this one. There was one guy who had something that resembled this problem, but in the end, it had something to do with the Elasticsearch, not Fluentd...

Kate Orlova
  • 3,225
  • 5
  • 11
  • 35
Lucas Coppio
  • 328
  • 2
  • 12
  • 1
    What is your buffer settings ? And Is ES getting timed out due too many requests ? – Sanoob Jun 17 '18 at 16:06
  • no other setting than what you see here. ElasticSearch does not timeouts, the only appearent offender here is FluentD container, which simply stops accepting packates/logs after this 3hours – Lucas Coppio Aug 25 '18 at 18:43
  • Did you come to any kind of solution to this? I'm having a similar issue. Mine eventually recovers a few hours later though. – Seer Nov 22 '18 at 12:00
  • Hi @Seer, its been a few months, but no, I had not found any way to fix it. Instead I`m using the fluentD in k8s as a sidecar on my pods using the tail plugin, and by some reason it is working just fine... so... yeah, I guess that moving to k8s fixed the problem? – Lucas Coppio Mar 01 '19 at 15:04

0 Answers0