Stack Overflow
Stack Overflow

Questions tagged [fluent-bit]

Fluent-Bit (not to be confused with Fluentd), is an open-source, light weight data collector which can act as both an aggregator and forwarder. It allows you to collect data/logs from different sources, unify and send them to multiple destinations.

Fluent bit links :

379 questions
14
votes
2 answers

Fluent-bit - Splitting json log into structured fields in Elasticsearch

I am trying to find a way in Fluent-bit config to tell/enforce ES to store plain json formatted logs (the log bit below that comes from docker stdout/stderror) in structured way - please see image at the bottom for better explanation. For example,…
BentCoder
  • 12,257
  • 22
  • 93
  • 165
11
votes
1 answer

Use Release.Name in values.yaml in helm

I am trying to use stable/fluent-bit as a sub-chart in my chart. That chart has a value in values.yaml: backend: es: host: elasticsearch How can I set the value of backend.es.host as something like {Release.Name}-elasticsearch without making…
Nsen
  • 281
  • 1
  • 3
  • 8
11
votes
5 answers

How to exclude namespace from fluent-bit logging

Is there a way to exclude certain namespaces in fluent-bit? I would like to exclude certain namespaces, so that fluent-bit doesn't forward all logs created in those namespaces to ELK. Is there a way to do it besides adding annotation to each pod in…
Oren
  • 113
  • 1
  • 1
  • 7
9
votes
1 answer

Can not get to work Fluentbit multi-line parser in k8s env

My project is deployed in k8s environment and we are using fluent bit to send logs to ES. I need to send java stacktrace as one document. Therefore I have used fluent bit multi-line parser but I cannot get it work. Approach 1: As per lot of…
Viraj
  • 5,083
  • 6
  • 35
  • 76
8
votes
1 answer

Log entries lost while using fluent-bit with kubernetes filter and elasticsearch output

From time to time we find that some logs are missing in the ES, while we are able to see them in Kubernetes. Only problems in logs I was able to find, point out to a problem with the kubernetes parser with things like these in the fluent-bit…
carrotcakeslayer
  • 809
  • 2
  • 9
  • 33
8
votes
1 answer

Serilog logs collected by Fluentbit to Elasticsearch in kubernetes doesnt get Json-parsed correctly

Using the EFK Stack on Kubernetes (Minikube). Have an asp.net core app using Serilog to write to console as Json. Logs DO ship to Elasticsearch, but they arrive unparsed strings, into the "log" field, this is the problem. This is the console…
imbageek
  • 609
  • 5
  • 18
7
votes
0 answers

Kubernetes add timestamp into every docker container log entry

I have 2 cluster Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-eks-f17b81", GitCommit:"f17b810c9e5a82200d28b6210b458497ddfcf31b", GitTreeState:"clean", BuildDate:"2021-10-15T21:46:21Z", GoVersion:"go1.15.15",…
Tho Quach
  • 1,347
  • 10
  • 26
7
votes
3 answers

Can fluent-bit parse multiple types of log lines from one file?

I have a fairly simple Apache deployment in k8s using fluent-bit v1.5 as the log forwarder. My setup is nearly identical to the one in the repo below. I'm running AWS EKS and outputting the logs to AWS ElasticSearch…
chakatz
  • 312
  • 1
  • 2
  • 10
6
votes
1 answer

Exclude pattern on a grep filter on fluent-bit does not seem to be working

I am trying to filter out a few records from the tail input to fluent-bit. But that does not seem to work. From the log files I need to exclude from all records with key value 'log' 1) Records that have 1 or more digits followed by a space 2)…
Sudeep Amin
  • 101
  • 2
  • 5
6
votes
2 answers

Fluentbit Kubernetes - How to extract fields from existing logs

I have configured EFK stack with Fluent-bit on my Kubernetes cluster. I can see the logs in Kibana. I also have deployed nginx pod, I can see the logs of this nginx pod also in Kibana. But all the log data are sent to a single field "log" as shown…
karthikeayan
  • 4,291
  • 7
  • 37
  • 75
6
votes
0 answers

Log collection from Android devices

I've been reading recently about Fluentd and Fluent-bit as tools for log unifying and collection. The documentation says it supports a few Linux distributions but I couldn't find any reference to Android - either that it is supported or not…
DannyL
  • 505
  • 4
  • 10
5
votes
1 answer

Update: Fluent bit parsing JSON log as a text

I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them {"@timestamp":"2022-05-06T06:02:10.669Z", "log.level": "INFO", "message":"INFO Health check ok", "ecs.version":…
Raushan
  • 307
  • 3
  • 12
5
votes
1 answer

How do you send Fargate EKS Fluent Bit Logs from different services to separate Cloudwatch groups?

I have followed this guide to configure Fluent Bit and Cloudwatch on my EKS cluster, but currently all of the logs go to one log group. I tried to follow a separate tutorial that used a kubernetes plugin for Fluent Bit to tag the services before the…
Frederick Haug
  • 245
  • 2
  • 14
5
votes
2 answers

Elasticsearch Dynamic Field Mapping and JSON Dot Notation

I'm trying to write logs to an Elasticsearch index from a Kubernetes cluster. Fluent-bit is being used to read stdout and it enriches the logs with metadata including pod labels. A simplified example log object is { "log": "This is a log…
rpf3
  • 651
  • 1
  • 10
  • 21
5
votes
2 answers

How can fluent-bit add custom metadata to each event message being sent to splunk

I'm using fluent-bit within Kubernetes to forward logs to Splunk. We'll be using the same Splunk index for multiple Kubernetes clusters, so I want to tag each event being forwarded from fluent-bit with the cluster that it comes from. I tried using…
truncj
  • 121
  • 1
  • 6
1
2 3
…
25 26