Unable to connect to SSL enabled Elastic Search from Google Dataflow

Question

Unable to connect to SSL enabled Elastic Search using Google Cloud Dataflow. I have used google provide template from git I modified the source code to pass keystore, keystorepassword, truststore to the job along with the Elastic Search credentials

Elastic Search Class file

config = config.withKeystorePassword("XXXXXXXXXXXXXXXXXXXXXXXXXXX")
        .withKeystorePath("PATH_TO_KEYSTORE")

Exception on running the job is as below

Caused by: java.lang.IllegalArgumentException: Cannot get Elasticsearch version at com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1546) at com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336) Caused by: org.elasticsearch.client.ResponseException: method [GET], host [https://elastic.irds.opus.dev.gcp.db.com], URI [/], status line [HTTP/1.1 401 Unauthorized] {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) at com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1530) at com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:1336) at com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown Source)

We wrote a sample java program to ensure elastic search connectivity, which works fine as expected

SSLContextBuilder sslBuilder = SSLContexts.custom()
        .loadKeyMaterial(keyStore,"*************".toCharArray())
        .loadTrustMaterial(truststore,null);
final SSLContext sslContext = sslBuilder.build();
RestClientBuilder builder = RestClient.builder(
        new HttpHost("https://X.X.X.X", 443, "https"))
        .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            public HttpAsyncClientBuilder customizeHttpClient(
                    final HttpAsyncClientBuilder httpAsyncClientBuilder) {
                httpAsyncClientBuilder.disableAuthCaching();
                return httpAsyncClientBuilder.setSSLContext(sslContext);
            }
        });

RestHighLevelClient client = new RestHighLevelClient(builder);

GetRequest getRequest = new GetRequest("test_index1/_search", "1");
GetResponse response = client.get(getRequest, RequestOptions.DEFAULT);
System.out.println("\nSearch results:");
System.out.println(response.getSourceAsString());

Could you please provide some inputs on what changes do i need to make on dataflow template to connect to SSL enabled Elastic search

Unable to connect to SSL enabled Elastic Search from Google Dataflow

0 Answers0