fluentd elasticsearch plugin - The client is unable to verify that the server is Elasticsearch

Question

I want to send some nginx logs from fluentd to elasticsearch , however, fluentd is unable to start due to following error message:

The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.

[error]: #0 unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not Elasticsearch and we do not support this unknown product."

This is my fluentd config :

<source>
  @type tail  
    <parse>    
      @type nginx 
    </parse>  
  path /tmp/lab4/nginx/access.log  
  pos_file /tmp/lab4/nginx/access.po
  tag nginx.access
</source>

<match nginx.**>
 @type elasticsearch
 scheme http
 host 192.168.1.154 
 port 9200 
 with_transporter_log true
 @log_level debug
</match>

If I do a curl http://192.168.1.154:9200 , I can see a response from Elasticsearch with the system version and other info .

For reference I am using :

fluentd version 1.14.5
fluentd elastic-search-plugin 5.2.0
elastic-search 7.12.0

Any idea on what I am doing wrong ?

In the above snapshot, the [elasticsearch](https://github.com/elastic/elasticsearch-ruby) client gem version (used by `fluent-plugin-elasticsearch`) is 8.0.0. You are using ElasticSearch v7.12.0 which is evaluated as unsupported. See: https://github.com/elastic/elasticsearch-ruby/blob/ce84322759ff494764bbd096922faff998342197/elasticsearch/lib/elasticsearch.rb#L110-L119. So, it looks like you need to install an equivalent supported version. — Azeem, Feb 26 '22 at 06:49
thanks for pointing out the validation, I tried upgrading my Elasticsearch to version 8 , but couldn't make it work ... In the end, what I did is upgrade Elasticsearch to version 7.17 ,and downgrade my fluentd Elasticsearch plugin to v 7.12 `sudo fluent-gem install elasticsearch -v 7.12` — rugby2312, Feb 26 '22 at 07:50
You're welcome! Glad you could make it work! You might want to post a detailed answer on how you made it work if someone else encounters this issue. — Azeem, Feb 26 '22 at 08:20
Thanks @Azeem — your comment is very helpful, you should post an answer, you deserve the rep! — Darragh Enright, Apr 05 '22 at 12:04
@DarraghEnright: Glad it was helpful! :) Answer posted. Thank you! — Azeem, Apr 05 '22 at 15:32

score 5 · Answer 1 · answered May 15 '22 at 14:35

for anyone who is facing the issue in docker, the below steps solved the issue for me:

need to build the fleutd with the "elasticsearch gem" as per the version of the elasticsearch being used, like below: Dockerfile:

FROM fluent/fluentd
RUN gem install elasticsearch -v 7.6
RUN gem install fluent-plugin-elasticsearch
RUN gem install fluent-plugin-rewrite-tag-filter
RUN gem install fluent-plugin-multi-format-parser

Mention the es version in the out plugin of es in fluent.conf:

@type elasticsearch
host 10.10.13.21
port 9200
verify_es_version_at_startup false
default_elasticsearch_version 7

score 3 · Answer 2 · answered Apr 05 '22 at 15:31

In that snapshot, the elasticsearch client gem version (used by fluent-plugin-elasticsearch) is 8.0.0. You are using ElasticSearch v7.12.0 which is evaluated as unsupported.

See https://github.com/elastic/elasticsearch-ruby/blob/ce84322759ff494764bbd096922faff998342197/elasticsearch/lib/elasticsearch.rb#L110-L119.

So, it looks like you need to install an equivalent supported version.

fluentd elasticsearch plugin - The client is unable to verify that the server is Elasticsearch

2 Answers2