splitting docker stdout and stderr with fluentd fluent-plugin-rewrite-tag-filter plugin

Question

I currently have the following config:

<match docker.nginx>
  @type rewrite_tag_filter
  rewriterule1 source stdout docker.nginx.stdout
  rewriterule2 source stderr docker.nginx.stderr
</match>

but this means, that with each container I have to do the same.

This isn't working, but probably you'll get what I want from it:

<match docker.*>
  @type rewrite_tag_filter
  rewriterule1 source stdout docker.*.stdout
  rewriterule2 source stdout docker.*.stderr
</match>

So my question is can I somehow refer to the matched tag in the match block? So if it's nginx/rabbitmq/zookeeper/anything, it will split all event flows into docker.<fluentd-tag>.stdout and stderr.

Thanks in advance!

nah, sorry :(. What I've seen how kubernetes does the splitting is to send the log entries into files and then fluentd parsing them. — csaboable, Jul 07 '19 at 08:57

score 1 · Accepted Answer · answered Aug 08 '23 at 18:35

Looking at Fluentd record_transformer parameter , you could try and use placeholders to refer to captured patterns in the tag, applying dynamic transformations based on those captured values.

For your requirement, you would use a wildcard (*) to capture the tag and then refer to that captured value using a placeholder (${tag_parts[N]}) in the rewrite_tag_filter plugin.

<match docker.*>
  @type rewrite_tag_filter
  rewriterule1 source stdout docker.${tag_parts[1]}.stdout
  rewriterule2 source stderr docker.${tag_parts[1]}.stderr
</match>

In this case, ${tag_parts[1]} would refer to the part of the tag after docker. (e.g., nginx, rabbitmq, zookeeper, etc.).

Any tag that matches the pattern docker.* will have its events rewritten based on the source (either stdout or stderr) to tags like docker.nginx.stdout, docker.rabbitmq.stderr, and so on.

splitting docker stdout and stderr with fluentd fluent-plugin-rewrite-tag-filter plugin

1 Answers1