How to connect 'spring boot 2.1 with elasticsearch 6.6' with cluster node 'https'?

Question

I'm working on a new project which involves fuzzy search so was working on elasticsearch (v6.4.3) with spring boot (v2.1.5). I am unable to make connection between spring boot to elasticsearch since I have to pass username, password, ca_certificate_base64 from spring boot to elasticsearch to make connection. Could you please let me know how do I make connection and what elasticsearch client is correct option? A sample or link would be helpful.

Locally (my laptop), I did install elasticsearch 6 and spring boot 2.1.5. I was able to make connection (using spring data) since there was no 'https' connection required or pass username and password since most of the examples in internet talks about localhost:9200.

   public Client client() {
     Settings elasticsearchSettings = Settings.builder()
           .put("client.transport.sniff", true)
           .put("uri", "https://usertemp:ABCSAD@alp-usba-north-3-portal.11.db2lay.com:32117/")
           .put("uri_direct_1", "https://usertemp:ABCSAD@alp-usba-north-3-portal.11.db2lay.com:32117/")
           .put("cluster.name", clusterName).build();
      TransportClient client = new PreBuiltTransportClient(elasticsearchSettings);
}


I also tried to update application.properties file
'''
spring.data.elasticsearch.cluster-name=ee842f-93042
spring.data.elasticsearch.cluster-nodes=alp-usba-north-3-portal.11.db2lay.com:32117
spring.data.elasticsearch.properties.username=usertemp
spring.data.elasticsearch.properties.password=ABCSAD
spring.data.elasticsearch.properties.ca_certificate_base64=SUZJQ0FURS0tLS0tCk1JSURvekNDQW91Z0F3SUJBZ0lFWFA5Sjl6QU5CZ2

I was expecting the connection with spring boot and elasticsearch. But, getting below error when I start the application server.

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.transport.TransportClient]: Factory method 'elasticsearchClient' threw exception; nested exception is java.lang.IllegalArgumentException: unknown setting [password] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    ... 96 common frames omitted
Caused by: java.lang.IllegalArgumentException: unknown setting [password] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:393) ~[elasticsearch-6.4.3.jar:6.4.3] 

Answer 1

As I already mentioned in the comments, I don't know how exactly IBM Compose for Elasticsearch sets up the connection security; my setup for this scenario is as follows:

Elasticsearch

• Elasticsearch 6.6.2 running on my machine
• an NGINX running as proxy in front of Elasticsearch with
  • Basic Authentication activated
  • SSL only
  • SSL client certificates needed

maven and versions

In order to be able to communicate via HTTPS you need to use the RestClient which is available in Spring Data Elasticsearch as of version 3.2 (which currently is available in version 3.2.0.M4). Spring Boot 2.1.5 pulls in Spring Data Elasticsearch 3.1.8, so you need to override the version. You also need to specify the Elasticsearch version to match Spring Boot Data 3.2.0.M4, so your pom.xml needs the following entries:

<properties>
    <java.version>1.8</java.version>
    <elasticsearch.version>6.7.2</elasticsearch.version>
    <spring-data-elasticsearch>3.2.0.M4</spring-data-elasticsearch>
</properties>

<repositories>
    <repository>
        <id>spring-libs-snapshot</id>
        <name>Spring Snapshot Repository</name>
        <url>https://repo.spring.io/libs-milestones</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-elasticsearch</artifactId>
        <version>${spring-data-elasticsearch}</version>
    </dependency>
</dependencies>

Bean configuration

In your program you can configure the Beans for Spring Data Elasticsearch with an implementation of AbstractElasticsearchConfiguration:

@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {

    private static final Logger LOG = LoggerFactory.getLogger(RestClientConfig.class);

    private static final String CERT_FILE = "client.p12";
    private static final String CERT_PASSWORD = "topsecret";
    private static final String USER_NAME = "user";
    private static final String USER_PASS = "password";

    @Override
    public RestHighLevelClient elasticsearchClient() {

        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo("localhost:443")  // set the address of the Elasticsearch cluster
                .usingSsl(createSSLContext())  // use the SSLContext with the client cert
                .withBasicAuth(USER_NAME, USER_PASS)   // use the headers for authentication
                .build();

        return RestClients.create(clientConfiguration).rest();
    }

    private SSLContext createSSLContext() {
        try {
            SSLContext sslContext = SSLContext.getInstance("TLS");

            KeyManager[] keyManagers = getKeyManagers();

            sslContext.init(keyManagers, null, null);

            return sslContext;
        } catch (Exception e) {
            LOG.error("cannot create SSLContext", e);
        }
        return null;
    }

    private KeyManager[] getKeyManagers()
            throws KeyStoreException, NoSuchAlgorithmException, IOException, CertificateException, UnrecoverableKeyException {
        try (InputStream inputStream = getClass().getClassLoader().getResourceAsStream(CERT_FILE)) {
            KeyStore clientKeyStore = KeyStore.getInstance("PKCS12");
            clientKeyStore.load(inputStream, CERT_PASSWORD.toCharArray());

            KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
            kmf.init(clientKeyStore, CERT_PASSWORD.toCharArray());
            return kmf.getKeyManagers();
        }
    }

    @Bean
    @Primary
    public ElasticsearchOperations elasticsearchTemplate() {
        return elasticsearchOperations();
    }
}

With this setup I can run my Spring Boot application against the Elasticsearch cluster using ElasticsearchRestTemplate and ElasticsearchRepository.

You probably need to adjust the code in the getKeyManagers() method to match what you have from Compose, but this should give you some point from where to start.