Connect aws-elasticssearch with fluentd using tdagent

Question

I have connected elasticsearch on local machine successfully using tdagent, but in staging enviorment I need to connect with aws elasticsearch, If I have installed fluentd using ruby, then there is a plugin

gem 'fluent-plugin-aws-elasticsearch-service'

But when using fluentd with tdagent how we can configure aws-elasticsearch?

<match catalog>
  @type elasticsearch
  host https://elastic_devel_aws.com
  # port 9200
  logstash_format true 
  include_timestamp true 
  index_name _logs_test
  flush_interval 10s
  reconnect_on_error true
  reload_on_failure true
  reload_connections false
  request_timeout 120s
  <buffer>
    @type file
   flush_interval 10s
   retry_type periodic
   retry_forever true
   retry_wait 10s
   chunk_limit_size 16Mb
   queue_limit_length 4096
   total_limit_size 60Gb
   path /var/lib/td-agent/buffers/output_elasticsearch-1
 </buffer>
</match>

When tried with something like this getting

  Could not communicate to Elasticsearch, resetting connection and trying again. getaddrinfo: Name or service not known (SocketError)

score 1 · Answer 1 · answered Feb 23 '20 at 22:37

To install the gem with td-agent you need to execute the following command

/usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service'

and you can use the following configs to send the logs to AWS managed elastcisearch.

<match *.**>
  @type elasticsearch

  host "#{ENV['ELASTICSEARCH_HOST']}"
  port "#{ENV['ELASTICSEARCH_PORT']}"
  scheme "#{ENV['ELASTICSEARCH_SCHEME'] || 'http'}"
  type_name log

  include_tag_key true
  tag_key @logtag

  logstash_format true
  logstash_prefix fos
  logstash_dateformat %Y%m%d

  reconnect_on_error true

  <buffer>
    @type file
    path /fluentd/log/fos/elastic-buffer
    flush_thread_count 8
    flush_interval 1s
    chunk_limit_size 32M
    queue_limit_length 4
    flush_mode interval
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

You also need to make sure that the staging environment is allowed to reach the elastic search instance, you can check this by executing the following command

telnet elastic_devel_aws.com 443

Connect aws-elasticssearch with fluentd using tdagent

1 Answers1