1

I am setting up log forwarding for my Linux & Windows machines on a RHEL8 VM using Fluentd and Elasticsearch. This is what I have installed:

  • RHEL8 - VMWare VM
  • Fluentd v1.0
  • td-agent 4.5.0 fluentd 1.16.1
  • fluent-plugin-elasticsearch-5.3.0

I am able to redirect the logs to the td-agent.log file but not able to forward the logs to Elasticsearch.

After installing the fluent-plugin-elasticsearch plugin, when I turn on the td-agent, it continuously generates these logs:

2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="/var/log/td-agent/td-agent.log" rotate_age=nil rotate_size=nil
2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog"
2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch"
2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in rescue in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in method_missing' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:in info'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in detect_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in block in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in retry_operate' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in add_match' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in block in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in each' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in run_configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:in block in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in main_process' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in <top (required)>'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in load' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in 
2023-06-14 04:45:26 -0500 [error]: Worker 0 exited unexpectedly with status 1

This is my td-agent.conf configuration for syslog forwarding:

<source>
  @type syslog
  port 5140
  tag system
</source>

<match system.**>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
</match>

My objective is to forward the logs to Elasticsearch and use Kibana dashboard to filter and analyze.

I have already tried using a lower version on the fluentd-elasticsearch plugin but at that time the agent did not start at all. Have tried the same conf in another non-hardened RHEL7 machine.

Azeem
  • 11,148
  • 4
  • 27
  • 40
Manish
  • 21
  • 3
  • Where is ElasticSearch deployed? On the same machine? Are you able to connect on `localhost:9200` on the same machine e.g. with `telnet`, `curl`, etc.? – Azeem Jun 15 '23 at 07:42
  • Here are already reported issues: https://github.com/uken/fluent-plugin-elasticsearch/issues/139 and https://github.com/uken/fluent-plugin-elasticsearch/issues/912 – Azeem Jun 15 '23 at 07:43
  • Yes I can connect on 9200 port. Both fluentd & elasticsearch are on the same machine: # telnet 127.0.0.1 9200 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. Connection closed by foreign host. # curl -i http://127.0.0.1:9200/ curl: (52) Empty reply from server # /bin/systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2023-06-13 10:32:36 CDT; 1 day 21h ago Main PID: 1499 (java) – Manish Jun 15 '23 at 13:10
  • Right. Have you looked at above issues? Do you find anything useful there? – Azeem Jun 15 '23 at 13:18
  • 1
    Not exactly. But the curl output helped me find this: https://stackoverflow.com/questions/35921195/curl-52-empty-reply-from-server-timeout-when-querying-elastiscsearch I disabled the xpack security in ES configuration and it worked. In /etc/elasticsearch/elasticsearch.yml2, changed xpack.security.enabled: false Restarted ES service and the errors in td-agent were gone. Thanks @Azeem for your help – Manish Jun 15 '23 at 17:01
  • Awesome! You might want to self-answer this with these details. Would definitely be helpful for others. – Azeem Jun 15 '23 at 17:10

1 Answers1

1

I found the solution in a different thread:

"curl: (52) Empty reply from server" / timeout when querying ElastiscSearch

What made me search this thread was the Curl output:

    # curl -i -X --HEAD http://127.0.0.1:9200/
    curl: (52) Empty reply from server

Changed Xpack security parameter in /etc/elasticsearch/elasticsearch.yml2 from true to false:

    xpack.security.enabled: false

Restarted ES service and the errors in td-agent logs were gone.

Manish
  • 21
  • 3