create index for elasticsearch as namespaces names

Question

im useing elasticsearch opendistro whith fluentd and i want to collect my kubernetes cluster logs , i want collect logs per namespace in index's . im lookin this answer but still having problem.also i added Fluentd-${record['kubernetes']['namespace_name']} but it couldn't defined my namespaces.

im using this conf for source

    ## logs from podman
    <source>
      @type tail
      @id in_tail_container_logs
      @label @KUBERNETES
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>

and about filters.conf

 <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>


      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
      </filter>

      <filter kubernetes.**>
        @id filter_parser
        @type parser
        key_name log
        reserve_data true
        remove_key_name_field true
        <parse>
          @type multi_format
          <pattern>
            format json
          </pattern>
          <pattern>
            format none
          </pattern>
        </parse>
      </filter>

      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>

and finally in output

  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host myhost
        port 9200
        user myuser
        password mypass
        scheme https
        ssl_verify false
        logstash_prefix Fluentd-${record['kubernetes']['namespace_name']}
        logstash_format true
        <buffer tag, $.kubernetes.namespace_name>
            flush_thread_count 8
            flush_interval 5s
            chunk_limit_size 2M
            queue_limit_length 32
            retry_max_interval 30
            retry_forever true
        </buffer>
      </match>
    </label>

but in index still i haven't anything

Index names can only have lowercase letters, so if your prefix start with `Fluentd-`, it will not work, change it to `fluentd-` and try again. Also there are other restrictions for the naming, you can check it in the [opendistro documentation](https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/index-data/#naming-restrictions-for-indices). — leandrojmp, Oct 29 '21 at 15:21
thanks , i replace it with fluentd and upgrade fluentd with helm now on elastic in Index Management/Indices i have indexnames like this [fluentd-MYNAMESPACEs-2021.10.29 ] . but when im going to index pattern page in kibana for creating index i just have indexs exact like this [fluentd-${record['kubernetes']['namespace_name']}-2021.09.25 ],now how can i create those indices as index in discover page? — zone sd, Oct 29 '21 at 16:46
You mean that you have an index with this literal name `fluentd-${record['kubernetes']['namespace_name']}-2021.09.25`? This could mean that the value for `${record['kubernetes']['namespace_name']}` does not exist, but I do not use fluentd, so I can not help further. — leandrojmp, Oct 29 '21 at 17:13
yes in indices page i have fluentd-${record['kubernetes']['namespace_name']}-2021.09.25 name and still i cant create index with this,, ok thanks — zone sd, Oct 29 '21 at 19:08

score 3 · Answer 1 · answered Aug 24 '22 at 12:47

I was recently working on a fluent-bit -> fluentd -> opensearch setup so just putting my solution here.

In my case, I was also getting the literal ${record['kubernetes']['namespace_name']} as my index instead of the actual namespace (tried different variations like accessor pattern, with or without quotes, double/single etc but didn't work). If you do not need the tag, you can use it to pass the index name by rewriting it:

<match kube.**>
  @type rewrite_tag_filter
  <rule>
    key $['kubernetes']['namespace_name']
    pattern ^(.+)$
    tag $1
  </rule>
</match>

And on your output,

logstash_prefix fluentd-${tag}
logstash_format true

Hope it helps even though this can be considered a hack.

score 1 · Answer 2 · answered Jun 28 '23 at 09:04

You are using the wrong format to specify the namespace. Instead of ${record['kubernetes']['namespace_name']} you must use ${$.kubernetes.namespace_name}. It's also documented here.

I got the correct solution from this issue thread, the output part should look something like this:

      logstash_format true
      logstash_prefix Fluentd-${$.kubernetes.namespace_name}
      <buffer tag, $.kubernetes.namespace_name>
        @type memory
        timekey 5
        timekey_wait 5
      </buffer>

I hope this helps!

create index for elasticsearch as namespaces names

2 Answers2