Unable to access ElasticSearch AWS through Python

Question

I'm trying to access ElasticSearch AWS from my localhost through Python (I can access it through my browser).

from elasticsearch import Elasticsearch
ELASTIC_SEARCH_ENDPOINT = 'https://xxx'
es = Elasticsearch([ELASTIC_SEARCH_ENDPOINT])

I'm receiving this error:

ImproperlyConfigured('Root certificates are missing for certificate validation. Either pass them in using the ca_certs parameter or install certifi to use it automatically.',)

How can I access it? I have not configured any certificate, I only liberated the IPs that can access ElasticSearch Service.

score 21 · Answer 1 · answered Dec 12 '17 at 17:11

for python 3.5 install certifi and use ca_certs=certifi.where() this will pass the certificates

import certifi
from elasticsearch import Elasticsearch

host = 'https://###########.ap-south-1.es.amazonaws.com'

es = Elasticsearch([host], use_ssl=True, ca_certs=certifi.where())

score 8 · Accepted Answer · answered Mar 28 '17 at 10:09

elasticsearch-py doesn’t ship with default set of root certificates. To have working SSL certificate validation you need to either specify your own as ca_certs or install certifi which will be picked up automatically.

from elasticsearch import Elasticsearch

# you can use RFC-1738 to specify the url
es = Elasticsearch(['https://user:secret@localhost:443'])

# ... or specify common parameters as kwargs

# use certifi for CA certificates
import certifi

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True 
)

# SSL client authentication using client_cert and client_key

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True,
    ca_certs='/path/to/cacert.pem',
    client_cert='/path/to/client_cert.pem',
    client_key='/path/to/client_key.pem',
)

https://elasticsearch-py.readthedocs.io/en/master/

score 6 · Answer 3 · answered May 30 '17 at 22:25

I did it this way and it worked:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

host = 'YOURHOST.us-east-1.es.amazonaws.com'
awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')

es = Elasticsearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)
print(es.info())

kartik · Answer 4 · 2018-04-19T10:00:24.290

You can also use boto3 to generate tempory access key & secret key.

import boto3

region = 'ap-southeast-2'
service = 'es'
session = boto3.Session()
credentials = session.get_credentials()

awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,region, service,session_token=credentials.token)

es = Elasticsearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html

Unable to access ElasticSearch AWS through Python

4 Answers4