How do you send Fargate EKS Fluent Bit Logs from different services to separate Cloudwatch groups?

Question

I have followed this guide to configure Fluent Bit and Cloudwatch on my EKS cluster, but currently all of the logs go to one log group. I tried to follow a separate tutorial that used a kubernetes plugin for Fluent Bit to tag the services before the reached the [OUTPUT] configuration. This caused issues because Fargate EKS currently does not handle Fluent Bit [INPUT] configurations as per the bottom of this doc.

Has anyone encountered this before? I'd like to split the logs up into separate services.

Here is my current YAML file .. I added the parser and filter to see if I could gain any additional information to work with over on Cloudwatch.

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep On
        
  filters.conf: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        # Kube_Tag_Prefix kube.var.log.containers.
        Kube_URL https://kubernetes.default.svc:443
        Merge_Log On
        Merge_Log_Key log_processed
        Use_Kubelet true
        Buffer_Size 0
        Dummy_Meta true
  
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region us-east-1
        log_group_name fluent-bit-cloudwatch2
        log_stream_prefix from-fluent-bit-
        auto_create_group On

score 6 · Accepted Answer · answered Jun 24 '21 at 14:28

6

So I found out that it is actually simple to do this.

The default tag of input on fluent bit contains the name of the service you are logging from, so you can actually stack multiple [OUTPUT] blocks each using the wildcard operator around the name of your service . That was all I had to do to get the streams to get sent to different log groups. Here is my YAML for reference.

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *logger*
        region us-east-1
        log_group_name logger-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On
        
    [OUTPUT]
        Name cloudwatch_logs
        Match   *alb*
        region us-east-1
        log_group_name alb-fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On

answered Jun 24 '21 at 14:28

Frederick Haug

245
2
14

1

hi, what do you mean by "name of the service you are logging from" ? do you mean Kubernetes Service? – aldred Jul 06 '21 at 07:26
I meant to say the given name of the pod, which is in the metadata part of a yaml – Frederick Haug Jul 06 '21 at 12:59
ah thanks, i tried the name of the pod, it works. do you know other criterias that we can match with? e.g. namespace, etc2 – aldred Jul 06 '21 at 13:17
I haven't made any other attempts, it seems like if you are using the fargate version of EKS you want have as many options work with. Happy to hear you were able to get my approach working though. – Frederick Haug Jul 06 '21 at 15:04
1

turns out filtering by K8S metadata is still in the roadmap: https://github.com/aws/containers-roadmap/issues/1197 – aldred Jul 06 '21 at 23:40
Im trying to do the exact same thing, but how does *logger* and *alb* match to some logs? – Tony Oct 19 '21 at 18:01
They are coming from the pod names, if you can manage to edit the name of pods then this approach should work for you – Frederick Haug Oct 21 '21 at 12:37
Ah thanks!. I've been trying to route logs by a keyword in the log itself, but no luck. https://stackoverflow.com/questions/69635637/eks-fargate-fluent-bit-multiple-outputs – Tony Oct 21 '21 at 16:16
I'm not sure why, but this approach is not working for me at all. I have the Match tag as \*\* just like you have mentioned. But no luck. – Pavan Kumar Lekkala Jul 18 '22 at 12:07
1

This answer just saved me after days of looking through docs and github issues. Severely underrated appraoch! – data princess Jan 18 '23 at 16:07

How do you send Fargate EKS Fluent Bit Logs from different services to separate Cloudwatch groups?

1 Answers1