Stack Overflow
Stack Overflow

Questions tagged [td-agent]

Please use this tag for question regarding td-agent, a data collection daemon. https://support.treasuredata.com/hc/en-us/articles/360000687108-Overview-of-Server-Side-Agent-td-agent-

Arm Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. You can continuously import data using td-agent.

71 questions
7
votes
2 answers

Fluentd pods running via daemonset getting terminated with warnings on google container engine

I am trying to run fluentd as a daemonset on kubernetes cluster (GKE). The config is getting parsed successfully, then the plugins are receiving shutdown signal with few warn messages. There are no error messages. I tried increasing the verbosity…
drishti ahuja
  • 1,379
  • 2
  • 15
  • 26
2
votes
1 answer

Extracting and logging only value of a key in fluentd

I have an input file with JSON messages (line delimited). The format of the messages is as…
Ankit Rastogi
  • 597
  • 4
  • 14
2
votes
1 answer

How to encrypt fluentd SQL plugin password?

Because of security reasons, we can't keep SQL authentication in plain text, is there a way to hide or encrypt passwords? I am getting bad documentation and bad support from the plugin site. Unfortunately I can't keep this data in environment…
Mysterious288
  • 365
  • 5
  • 24
2
votes
0 answers

Fluentd configuration for ORACLE clob datatype

I'm using fluentd configuration to read data from a text file and it is pushed to the Oracle database, I have clob & nclob type of datatype when fluentd pushed data, the column is always null and I don't see any errors. I'm not sure how to resolve…
Mysterious288
  • 365
  • 5
  • 24
2
votes
1 answer

Connect aws-elasticssearch with fluentd using tdagent

I have connected elasticsearch on local machine successfully using tdagent, but in staging enviorment I need to connect with aws elasticsearch, If I have installed fluentd using ruby, then there is a plugin gem…
Mariya James
  • 935
  • 2
  • 9
  • 27
2
votes
1 answer

Starting td-agent for fluentd

How to start td-agent service for fluentd configuration file which is not in the default /etc/td-agent/td-agent.conf location? I tried with td-agent -c /home/charith/Documents/fdConfig/configuration.conf but not sure how to restart the service…
Charith_32
  • 145
  • 2
  • 10
2
votes
1 answer

How to get fluentd / td-agent TLS/SSL encryption for in_forward to work?

I'm following the steps to enable TLS/SSL encryption using td-agent and I cannot get the test to pass (https://docs.fluentd.org/v1.0/articles/in_forward#how-to-enable-tls/ssl-encryption): 1) Created the certs, $ openssl req -new -x509 -sha256 -days…
pragmatic_programmer
  • 3,566
  • 3
  • 29
  • 36
2
votes
2 answers

Unable to start fluentD (td-agent)

I have installed fluentD(td-agent) version 2 in CentOs 7. Now I'm unable to start td-agent. While starting td-agent it was throwing some error -- Unit td-agent.service has begun starting up. Feb 02 14:53:39 chninitefk01 fluentd[2282]: …
rolz
  • 591
  • 2
  • 11
  • 23
2
votes
1 answer

Fluentd not sending parsed logs to Mongo

I am using Fluentd to parse the logs and store the parsed log in MongoDB. My application is generating the following logs: [2018-01-25 17:50:22] 192.168.10.1 GET http://localhost.com/mypage html 0 Mozilla/5.0 200 132 Fluentd is parsing the logs…
Veer Shrivastav
  • 5,434
  • 11
  • 53
  • 83
1
vote
0 answers

When huge logs data is flushed, Getting below buffer overflow throw_exception and BufferOverflowError in fluent (td-agent version:1.6)

[warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"…
AbhishekChalke
  • 11
  • 3
1
vote
1 answer

fluentd: Not able to send syslogs to Elasticsearch using fluent-plugin-elasticsearch

I am setting up log forwarding for my Linux & Windows machines on a RHEL8 VM using Fluentd and Elasticsearch. This is what I have installed: RHEL8 - VMWare VM Fluentd v1.0 td-agent 4.5.0 fluentd 1.16.1 fluent-plugin-elasticsearch-5.3.0 I am able…
Manish
  • 21
  • 3
1
vote
0 answers

Td-agent rename and import older logs

A few days ago our main Fluentd aggregator instance was wrongfully re-created. Because of it, our td-agent buffer instances lost connection to it for ~12 hours and accumulated a lot of logs. After fixing the connectivity and some td-agent restarts,…
user3248750
  • 11
  • 2
1
vote
2 answers

Fluentd NewRelic output "time must be a Fluent::EventTime (or Integer): Float"

I'm encountering an issue that I don't understand, extract of config of td-agent: @type gelf tag gelf-nr @type json port 12201 @type copy @type stdout …
Mickael Brangeon
  • 79
  • 1
  • 7
1
vote
1 answer

How to use if condition in fluentd 0.12

I am using fluentd 0.12. I have got below logs - {"log":"2020-09-18 19:44:41 | INFO | timeleft| 5a | 364 - core - 1.0.0.SNAPSHOT | TrafficTo:blue-connectorprocess from: blue-connectorprocess"} {"log":"2020-09-18 19:44:41 | INFO | timeleft| 6a | …
Puja
  • 13
  • 3
1
vote
1 answer

Enabling Fluentd Log rotation

I am using fluentd with the tg-agent installation. And I observed my default td-agent.log file is growing without having any log rotation. I am using the following command to run the td-agent /etc/init.d/td-agent start And I found the following…
Eshan Sudharaka
  • 607
  • 2
  • 9
  • 16
1
2 3 4 5