Adding Kubernetes Metadata To Custom FluentD Non DaemonSet @tail Event

Question

I have a pod that runs Fluentd as a sidecar that collects the logs from another container (sample application) and the common volume have been mounted on both containers (volumeMounts).

Sample Application Code writes a UTC timestamps to a file

   containers:
    - name: sampleApplication
      image: ${DOCKER_IMAGE}:${DOCKER_TAG}
      args:
        - /bin/sh
        - -c
        - >
          i=0;
          while true;
          do
            echo "$i: $(date)" >> /var/log/1.log;
            i=$((i+1));
            sleep 1;
          done

Tail output of a /var/log/1.log,

2849: Wed Mar 18 03:12:01 UTC 2020
2850: Wed Mar 18 03:12:02 UTC 2020
2851: Wed Mar 18 03:12:03 UTC 2020
2852: Wed Mar 18 03:12:04 UTC 2020
2853: Wed Mar 18 03:12:05 UTC 2020
2854: Wed Mar 18 03:12:06 UTC 2020

FluentD Configuration:

fluent.conf: |-
  <source>
    @type tail
    path /var/log/1.log
    pos_file /var/log/1.log.pos
    refresh_interval 5
    rotate_wait 5
    time_format %Y-%m-%dT%H:%M:%S.%N%Z
    format json
    keep_time_key true
    tag fluentd_event
  </source>

  <match **>
    @type secure_forward
    self_hostname "#{ENV['HOSTNAME']}"
    shared_key ****12345678****
    secure yes
    ca_cert_path /tmp/fluentd/keys/ca_cert.pem
    ca_private_key_path /tmp/fluentd/keys/external_ca_key.pem
    ca_private_key_passphrase ocpsecureforward

    <server>
       # This server accepts the fluentD events
      host 100.100.100.23
      port 24284
    </server>
  </match>

The above configuration works for me well where I can visualize the tail events of fluentD @tail plugin in Kibana, but what I would also like to achieve is adding the basic kubernetes metadata to the tail events such as namespace_name, pod_name and container_name. I have configured the kubernetes_metadata plugin as,

<filter **>
 type kubernetes_metadata
 kubernetes_url "#{ENV['K8S_HOST_URL']}"
 cache_size "#{ENV['K8S_METADATA_CACHE_SIZE'] || '1000'}"
 watch "#{ENV['K8S_METADATA_WATCH'] || 'false'}"
 bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
 ca_file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 use_journal "#{ENV['USE_JOURNAL'] || 'nil'}"
 container_name_to_kubernetes_regexp '^(?<name_prefix>[^_]+)_(?<container_name>[^\._]+)(\.(? 
 <container_hash>[^_]+))?_(?<pod_name>[^_]+)_(?<namespace>[^_]+)_[^_]+_[^_]+$'

I did run through some blogs and learnt that kubernetes metadata can be filtered and added FROM the container log file but I am NOT running fluentD as a DaemonSet. I am intended to run it as a sidecar and just collect the application log file using @tail plugin and add basic kubernetes metadata to the event.

Can kubernetes_metadata filter be leveraged to achieve this?

Any luck? I'm currently trying facing same problem. – Pavel P.F. May 14 '21 at 09:28 — Pavel P.F., May 14 '21 at 09:28
@PavelP.F. Not yet. Have you hand any? – Jninja Aug 05 '21 at 19:20 — Jninja, Aug 05 '21 at 19:20
@Jninja have you found a solution for this? – Chen A. Aug 15 '21 at 08:04 — Chen A., Aug 15 '21 at 08:04

score 0 · Answer 1 · answered Apr 08 '22 at 09:03

An alternative way could be using "downward API" and putting metadata information into environment variables, then using them via "#{ENV['POD_NAME']}" in fluentd config.

downward API Reference: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/

Adding Kubernetes Metadata To Custom FluentD Non DaemonSet @tail Event

1 Answers1