0

I'm using fluentd docker (1.14.5) with OpenSearch plugin (1.0.2) with the following config:

<match **>
    @type opensearch
    logstash_format true
    include_tag_key true
    tag_key "@log_name"
    <endpoint>
        url https://MY_AWS_ENDPOINT
        region MY_AWS_REGION
    </endpoint>
</match>

My dockerfile installs the plugin:

FROM fluent/fluentd:v1.14-1
RUN sudo gem install fluent-plugin-opensearch

During startup of the container inside AWS Beanstalk (platform 3.14), we see this error over and over:

logger_1   | 2022-03-15 13:30:20 +0000 [error]: #0 unexpected error error_class=Aws::Errors::NoSuchProfileError error="Profile `default' not found in /root/.aws/credentials or /root/.aws/config"
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/shared_config.rb:367:in `validate_profile_exists'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/shared_config.rb:103:in `credentials'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/shared_credentials.rb:42:in `initialize'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:222:in `new'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:222:in `aws_credentials'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:601:in `block in client'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/faraday-1.10.0/lib/faraday/connection.rb:91:in `initialize'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/faraday-1.10.0/lib/faraday.rb:120:in `new'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/faraday-1.10.0/lib/faraday.rb:120:in `new'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/http/faraday.rb:72:in `__build_connection'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:166:in `block in __build_connections'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:158:in `map'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:158:in `__build_connections'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:69:in `initialize'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:624:in `new'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:624:in `client'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/opensearch_index_template.rb:62:in `host_unreachable_exceptions'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/opensearch_index_template.rb:71:in `rescue in retry_operate'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/opensearch_index_template.rb:69:in `retry_operate'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:497:in `handle_last_seen_os_major_version'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.2/lib/fluent/plugin/out_opensearch.rb:380:in `configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin.rb:187:in `configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/agent.rb:132:in `add_match'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/agent.rb:74:in `block in configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `each'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/root_agent.rb:149:in `configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/engine.rb:105:in `configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/engine.rb:80:in `run_configure'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:716:in `block in run_worker'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:968:in `main_process'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:708:in `run_worker'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.5/bin/fluentd:15:in `<top (required)>'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
logger_1   |   2022-03-15 13:30:20 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
logger_1   | 2022-03-15 13:30:20 +0000 [error]: Worker 0 finished unexpectedly with status 1

AFAIK, fluentd should not fail, according to the docs. Instead, it should use the EC2 role automatically, since no key and secret were specified in the config file. The instance's role is properly configured so OpenSearch allows access to it. Still, no luck.

Any ideas what's going on?

Sagi Mann
  • 2,967
  • 6
  • 39
  • 72

1 Answers1

0

I found a workaround for now. Seems like (based on the stack trace) the plugin doesn't expect AWS sdk to throw an exception when trying to lookup credentials on disk. So I ended up adding this to my docker (note the exact spacing and lack of secret entry, otherwise it doesn't work)

RUN f=/root/.aws/credentials &&\
    mkdir -p `dirname $f` &&\
    echo "[default]" > $f &&\
    echo "aws_access_key_id = " >> $f

This seems to cause the code to NOT throw and continue to the next line, where it picks up the credentials from the instance profile successfully. I've also opened an issue with the plugin, as it's not expected behavior.

Sagi Mann
  • 2,967
  • 6
  • 39
  • 72