Update: Fluent bit parsing JSON log as a text

Question

I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them

{"@timestamp":"2022-05-06T06:02:10.669Z", "log.level": "INFO", "message":"INFO Health check ok", "ecs.version": "1.2.0","service.name":"spring-boot-application","event.dataset":"spring-boot-application","process.thread.name":"http-nio-9001-exec-7","log.logger":"com.app.designer.rules.controller.RulesController","transaction.id":"15e36b5ef6cc69dd","trace.id":"fd275718e25c061d309954773985b101"}

In Kibana, i can see the logs

JSON log is a value of log key. But, we want JSON Log key value, as Field and Value Please suggest. fluent bit config map is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/designer-logs.log.json
        Multiline         On
        Parser_Firstline  multiline_pattern
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         multiline_pattern
        Format       regex
        Regex        ^\[(?<LogDate>\d{2}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}\+\d{4})\] (?<LogLevel>[^ ]*) * (?<Thread>\S+) (?<Username>[^\s]*?) (?<Logger>[A-Za-z0-9$_.]+) -(\r\n|\r|\n)*(?<Message>.*)
        time_key     LogDate
        time_format  %d-%m-%y %H:%M:%S.%L%z

I believe it's related to `fluentbit` and not `fluentd`. Please edit and add relevant tags to attract the right audience. From the `fluentd` experience, it looks like the problem may be solved if you add a JSON parser before sending the output to ES. You need to test that. — Azeem, May 09 '22 at 07:03
Done the edits, i think with fluentbit parser we can achieve that, but not sure how? — Raushan, May 09 '22 at 07:13
Relevant: https://docs.fluentbit.io/manual/pipeline/parsers/json and https://stackoverflow.com/questions/68257635/fluentbit-parse-json — Azeem, May 09 '22 at 07:22
Please add that to your question under "Update" heading for others. Maybe, someone might be able to suggest something. — Azeem, May 10 '22 at 04:39
@Raushan did you get it resolved? Having the same behavior with our application — Rayyan, Jul 07 '23 at 07:31

score 1 · Answer 1 · answered Jul 07 '23 at 09:03

I changed parser configuration from regular expression to json parser to make it work.

Please find fluentbit confgmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/app.log
        Parser            json_parser
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         json_parser
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped_utf8    log    do_next
        Decode_Field_As   json       log

in Kibana

Thanks, my case was change from json parser to regex parser due to containerd runtime. — Rayyan, Jul 08 '23 at 08:03
@Rayyan getting same issue with containerd runtime, requesting you to share regex parser snippet. Also, is "kubernetes" filter is no more required? — Ashish Kumar, Aug 30 '23 at 22:11
Same as the crio parser mentioned in https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html — Rayyan, Aug 31 '23 at 15:45
awesome, its working. Thanks @Rayyan Sharing snippet for others (in case given linked get modified in future). ``` [FILTER] Name parser Match * Key_name log Parser crio [PARSER] Name crio Format Regex Regex ^(?[^ ]+) (?stdout|stderr) (?P|F) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z ``` Snippet looks distorted in comment, please ensure to re-structure it's indentation while applying. — Ashish Kumar, Aug 31 '23 at 21:26

Update: Fluent bit parsing JSON log as a text

1 Answers1