4

I'm currently using JournalD + JournalBeat + Logstash as logging stack but I want to switch to using JournalD + FluentD.

I tried using https://github.com/fluent-plugin-systemd/fluent-plugin-systemd as a input for FluentD but it results in a low throughput of ~1000 log lines per second and I need to support at least 2000.

So now I'm trying with JournalD + FluentBit + FluentD, using Forward protocol between FluentBit + FluentD. With this stack, I'm able to reach a throughput of 5000 log lines per second, but results in out of order lines. Actually, the out of order seems to be in chunks.

Here is my FluentBit config:

[SERVICE]
    # Flush
    # =====
    # Set an interval of seconds before to flush records to a destination
    Flush       5

    Daemon       Off
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name            systemd
    Tag             host.*
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Read_From_Tail  true
    Path            /var/log/journal

[OUTPUT]
    Name          forward
    Require_ack_response true
    Match         *
    Host          127.0.0.1
    Port          24224

Here is my FluentD config:

    <source>
      @type forward
      @id input_forward
      tag docker.systemd
    </source>

    <match docker.systemd>
      @type copy
      <store>
        @type file
        @id out_file_docker
        path /file-logs/${$.CONTAINER_TAG}/%Y/%m/%d/${$.PRIORITY}
        append true
        <format>
          @type single_value 
          message_key MESSAGE
        </format>
        <buffer $.CONTAINER_TAG,$.PRIORITY,time>
          @type file
          path /var/log/fluentd/file-buffers/
          timekey 1d
          flush_mode interval
          flush_interval 10s
          flush_at_shutdown true
        </buffer>
      </store>
      <store>
        @type prometheus
        <metric>
          name fluentd_output_status_num_records_total
          type counter
          desc The total number of outgoing 
        </metric>
      </store>
    </match>

Additional details:

  • I'm running FluentD and FluentBit in docker containers with 4gb and 4096 CPU shares
  • Measurements of CPU usage are less than 20% for both services

Other things I tried:

  • Setting Mem_Buf_Limit in FluentBit to a 2MB which fixes the out of order, but results in a throughput of only 350 lines per second. If I use a larger buffer log lines get out of order again.
  • Setting FluentBit output to file results in log lines in order but I lose the capability to distribute logs in different files.
  • Using larger Flush interval in FluentBit results in bigger chunks out of order
  • Tried flush_thread_count in FluentD with no impact

Any ideas of any other settings/protocols I should try? Is there any other way to integrate Journal and FluentD?

----- EDIT ----

Looking at FluentBit logs using DEBUG, I see:

[2020/01/14 17:00:30] [trace] [upstream] destroy connection #52 to 127.0.0.1:24224
[2020/01/14 17:00:30] [trace] [upstream] destroy connection #100 to 127.0.0.1:24224

So it looks like forward output is using multiple threads. Is that expected?

Jonathan Erlich
  • 41
  • 3
  • Looking at FluentBit logs using DEBUG, I see: ``` [2020/01/14 17:00:30] [trace] [upstream] destroy connection #52 to 127.0.0.1:24224 [2020/01/14 17:00:30] [trace] [upstream] destroy connection #100 to 127.0.0.1:24224 ``` So it looks like forward output is using multiple threads. Is that expected? – Jonathan Erlich Jan 14 '20 at 17:30

1 Answers1

0

You may add the Offset_Key in the input plugin, that might solve your problem while querying the data, moreover fluentbit doesn't follow the orders.

Selva Raj
  • 1