Fluentd is open-source and distributed data collector, which receives logs in JSON format, buffers them, and sends them to other systems like Amazon S3, MongoDB, Hadoop, Loki(Grafana system) or other Fluentds.
The use case is this:
I've several java applications running which all have to interact with different (each one has a specific target) elasticsearch indices. For instance an application A uses the indices A,B,C of ElasticSearch to query and update.…
By reading the following post from 12factor I have come up with a question I'd like to check how you guys handle this.
Basically, an app should write directly to stdout/stderr. Is there anyway to redirect these streams directly to fluentd (not bound…
I am trying to find a way in Fluent-bit config to tell/enforce ES to store plain json formatted logs (the log bit below that comes from docker stdout/stderror) in structured way - please see image at the bottom for better explanation. For example,…
I can't get Loki to connect to AWS S3 using docker-compose. Logs are visible in Grafana but the S3 bucket remains empty.
The s3 bucket is public and I have an IAM role attached to allow s3:FullAccess.
I updated loki to v2.0.0 and changed the period…
I currently have the following config:
@type rewrite_tag_filter
rewriterule1 source stdout docker.nginx.stdout
rewriterule2 source stderr docker.nginx.stderr
but this means, that with each container I have to do…
How can we easily transform with fluentd( and plugins ) something like this
{
"remote": "87.85.14.126",
"city": "saint-hubert"
}
To this:
{
"geoip": {
"remote": "87.85.14.126",
"city": "saint-hubert"
}
}
Thank you
I have the following configuration in my docker-compose file:
fluentd:
build: ./fluentd
container_name: fluentd
expose:
- 24224
- 24224/udp
depends_on:
- "elasticsearch"
networks:
- internal
public-site:
…
I am trying to create a centralized logging system using fluentd for a docker environment. Currently, i able to send the docker log to fluentd using fluentd docker logging driver which is a much cleaner solution compare to reading the docker log…
I was wondering how to use env vars in the Fluentd config, I tried:
type elasticsearch
logstash_format true
logstash_prefix $ENV_VAR
host ***
port ***
include_tag_key true
tag_key _key
but it doesn't work, any idea?
I am trying to use stable/fluent-bit as a sub-chart in my chart. That chart has a value in values.yaml:
backend:
es:
host: elasticsearch
How can I set the value of backend.es.host as something like {Release.Name}-elasticsearch without making…
I'm planning to run flask through gunicorn at kubernetes. To get the logging right, I want to output all my logs in json.
Currently I'm testing with minikube and https://github.com/inovex/kubernetes-logging to have fluentd collecting the logs.
I…
I'm a bit confused at how to setup error reporting in kubernetes, so errors are visible in Google Cloud Console / Stackdriver "Error Reporting"?
According to documentation
https://cloud.google.com/error-reporting/docs/setting-up-on-compute-engine
we…
I have source:
@type tail
tag service
path /tmp/l.log
format json
read_from_head true
I would like to make several filters on it and match it to several outputs:
@type tail
tag service.pi2
…
Facing: fluentd log unreadable. it is excluded and would be examined next time
I have a simple configuration for fluentD daemon set running in kubernetes setup.
Fluentd version: fluentd-0.12.43
Below is my configuration.
@type tail
…
