2

I am using deepset/haystack and communicating with elastic search. Using OpenDistroElasticsearchDocumentStore method works fine with username,pasword access to aws elastic search. Doesnt seem to work with role based access when deployed in ec2. Please suggest me a solution to access aws elastic search using python elastic search package given a role access

janpio
  • 10,645
  • 16
  • 64
  • 107
Sharath
  • 21
  • 4

2 Answers2

1

Do you mean IAM based access on AWS like this? We just recently merged a feature that might help you here (#965). Please install the latest Haystack version from the master branch and try something along those lines:

import boto3

from requests_aws4auth import AWS4Auth
from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from elasticsearch import RequestsHttpConnection

host = '<vpc_host>'
port = 443
region = 'eu-central-1'
service = 'es'
 
credentials = boto3.Session().get_credentials()
aws4auth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
 
document_store = OpenDistroElasticsearchDocumentStore(host=host,
                                            port=port,
                                            aws4auth=aws4auth,
                                            # can't be used with default es client version used in e.g. aws sagemaker
                                            embedding_field=None,
                                            index="document")
Dharman
  • 30,962
  • 25
  • 85
  • 135
Malte
  • 895
  • 2
  • 8
  • 16
  • Hi, In this step credentials = boto3.Session().get_credentials() It seems to be working in case of user access key and secret key. Isnt working as expected in case of roles. Please help me out with this – Sharath Apr 20 '21 at 13:26
  • Can you please clarify what you mean by "case of roles"? Can you provide an example for plain (open distro) elasticsearch without Haystack that works in your setup? From the elasticsearch docs my understanding is that the above is the "best practice" for IAM on AWS (https://elasticsearch-py.readthedocs.io/en/v7.12.0/index.html?highlight=aws#running-on-aws-with-iam) – Malte Apr 22 '21 at 08:13
  • Hi, I seem to have resolved the issue related to roles. There's a new error __init__() got an unexpected keyword argument 'aws4auth' when am using OpenDistroElasticsearchDocumentStore method. Thank you – Sharath Apr 26 '21 at 12:25
0
from requests_aws4auth import AWS4Auth
from botocore.session import Session
credentials = Session().get_credentials()
auth = AWS4Auth(region='eu-west-1', service='es', refreshable_credentials=credentials)

This example shows how to construct an AWS4Auth instance with automatically refreshing credentials, suitable for long-running applications using AWS IAM assume-role. The RefreshableCredentials instance is used to generate valid static credentials per-request, eliminating the need to recreate the AWS4Auth instance when temporary credentials expire.

Source: https://github.com/tedder/requests-aws4auth#dynamic-sts-credentials-using-botocore-refreshablecredentials

This was merged into AWS4Auth in May 2021.

Yann Stoneman
  • 953
  • 11
  • 35