AWS ElasticSearch Service version 2.3 does not work with Jest Client: java.net.SocketTimeoutException

Question

Some time ago Amazon Web Services added possibility to choose version of ElasticSearch service 1.5 or 2.3.

I have been using older version and Jest client worked perfectly fine with AWS and with ElasticSearch in memory (unit tests). When I changed dependency versions some errors occurred.

Current versions:

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>1.5.2</version>
</dependency>
<dependency>
    <groupId>io.searchbox</groupId>
    <artifactId>jest</artifactId>
    <version>2.0.2</version>
</dependency>
<dependency>
    <groupId>vc.inreach.aws</groupId>
    <artifactId>aws-signing-request-interceptor</artifactId>
    <version>0.0.11</version>
</dependency>

I updated to:

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.3.5</version>
</dependency>
<dependency>
    <groupId>io.searchbox</groupId>
    <artifactId>jest</artifactId>
    <version>2.0.3</version>
</dependency>
<dependency>
    <groupId>vc.inreach.aws</groupId>
    <artifactId>aws-signing-request-interceptor</artifactId>
    <version>0.0.13</version>
</dependency>

As well I can see here io.searchbox : jest-parent : 2.0.3 the current version of ElasticSearch is 2.1.0.

The question is, is there any chance to achieve the compatibility of Jest Client and AWS Elastic Search service version 2.3?

When I run my code I can see plenty of errors:

java.net.SocketTimeoutException: Read timed out

This exceptions has been raised from almost all calls to ES, only call for indices returned properly 404 using IndicesExists.Builder.

ElasticSearch Configuration class:

public class ElasticSearchConfig {

private static final String ES_REGION = "us-east-1";
private static final String ES_SERVICE_PREFIX = "es";

@Bean
public JestClient jestClient(@Value("${es.endpoint}") String esEndpoint) {

    final JestClientFactory factory = getJestClientFactory();

    factory.setHttpClientConfig(new HttpClientConfig
            .Builder(esEndpoint)
            .multiThreaded(true)
            .build());

    return factory.getObject();
}

private JestClientFactory getJestClientFactory() {

    return new JestClientFactory() {

        @Override
        protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
            builder.addInterceptorLast(prepareInterceptor());
            return builder;
        }

        @Override
        protected HttpAsyncClientBuilder configureHttpClient(HttpAsyncClientBuilder builder) {
            builder.addInterceptorLast(prepareInterceptor());
            return builder;
        }
    };
}

private AWSSigningRequestInterceptor prepareInterceptor() {
    AWSSigner signer = new AWSSigner(createCredentialsProvider(), ES_REGION, ES_SERVICE_PREFIX, LocalDateTime::now);
    return new AWSSigningRequestInterceptor(signer);
}
}

Full stack from unit tests:

2016-08-26 11:29:52,711 ERROR p.c.b.c.p.e.ESRepository (main) -  - Something went wrong during creation of new index
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_92]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_92]
    at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_92]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_92]
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[httpcore-4.4.3.jar:4.4.3]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.jar:4.5]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.jar:4.5]
    at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:109) ~[jest-2.0.3.jar:?]
    at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:56) ~[jest-2.0.3.jar:?]
    at pl.c.b.core.persistance.es.ESRepository.createIndex(ESRepository.java:138) ~[classes/:?]
    at pl.c.b.core.persistance.es.ESRepository.createIndexIfNotExists(ESRepository.java:129) [classes/:?]
    at pl.c.b.core.persistance.es.ESRepository.<init>(ESRepository.java:61) [classes/:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_92]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_92]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_92]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_92]
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772) [spring-beans-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839) [spring-context-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538) [spring-context-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
    at it.server.TomcatServer.start(TomcatServer.java:15) [test-classes/:?]
    at it.CoreItTestServer.startTomcatAndMocks(CoreItTestServer.java:92) [test-classes/:?]
    at it.CoreItTestServer.start(CoreItTestServer.java:53) [test-classes/:?]
    at it.tests.AbstractCoreItTest.startServices(AbstractCoreItTest.java:152) [test-classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_92]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_92]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84) [testng-6.8.8.jar:?]
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564) [testng-6.8.8.jar:?]
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213) [testng-6.8.8.jar:?]
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138) [testng-6.8.8.jar:?]
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291) [testng-6.8.8.jar:?]
    at org.testng.SuiteRunner.run(SuiteRunner.java:254) [testng-6.8.8.jar:?]
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) [testng-6.8.8.jar:?]
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) [testng-6.8.8.jar:?]
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224) [testng-6.8.8.jar:?]
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149) [testng-6.8.8.jar:?]
    at org.testng.TestNG.run(TestNG.java:1057) [testng-6.8.8.jar:?]
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:293) [surefire-testng-2.17.jar:2.17]
    at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:84) [surefire-testng-2.17.jar:2.17]
    at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:91) [surefire-testng-2.17.jar:2.17]
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) [surefire-booter-2.17.jar:2.17]
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) [surefire-booter-2.17.jar:2.17]
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) [surefire-booter-2.17.jar:2.17]

If you use TransportClient or the [http client](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_initialization.html#_initialization) do you get the same error? — Andrei Stefan, Aug 25 '16 at 21:55
@AndreiStefan I can do any HTTP action. I think it is more related to JestClient — ruhungry, Aug 26 '16 at 09:00
You mention `SocketTimeoutException`s but do you have a complete stacktrace? Is there any command that succeeds or all of them fail? — Andrei Stefan, Aug 26 '16 at 09:02
@AndreiStefan I edited the post and pasted full stack. Checking if index exists seems to work: `Checking if index already exist. JsonString: null, ErrorMessage: 404 Not Found, ResponseCode: 404, Succeeded: false`. Index does not exist, so 404 is returned. Creating a mapping, indexing a document, refreshing an index and searching for documents - all of them throw `java.net.SocketTimeoutException: Read timed out`. — ruhungry, Aug 26 '16 at 09:47
@AndreiStefan the same error occurred when I deployed my application to AWS EC2. Checking if index exists does not throw any error, so the application is able to connect with ES. Using Postman I was able to create new index. — ruhungry, Aug 26 '16 at 10:23
How do you check an index exists? If there is a connections issue because of some config mistake, you shouldn't be able to run anything on the cluster — Andrei Stefan, Aug 26 '16 at 10:42
@AndreiStefan here is a piece of code: `private void createIndexIfNotExists() { try { JestResult result = jestClient.execute(new IndicesExists.Builder(indexName).build()); logResult("Checking if index already exist.", result); if (result.getResponseCode() == NOT_FOUND_STATUS) { createIndex(); } } catch (IOException e) { log.error("Something went wrong during creation of new index", e); } }` It was working fine with ES version 1.5 and still does not throw any errors. — ruhungry, Aug 26 '16 at 10:59
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/121923/discussion-between-ruhungry-and-andrei-stefan). — ruhungry, Aug 26 '16 at 11:48
If you don't sign the http request, what happens? What kind of error do you get? — Andrei Stefan, Aug 27 '16 at 05:24
We are seeing a similar error using Jest and AWS ES v2.3. Were you able to figure out a way to fix this? I'm wondering if we need to configure the Jest maxConnectionIdleTime to enable the IdleConnectionReaper. — delux247, Sep 13 '16 at 15:29
@delux247 I haven't fixed it yet. I am not in hurry to update to 2.3, so I will probably wait for an update of JestClient — ruhungry, Sep 14 '16 at 08:06
@ruhungry I just tried connecting AWS ES 2.3 using ES 2.3.1 and Jest library that I downloaded today from github. All queries I tried worked flawlessly. I haven't tried creating an index, but search APIs seem to work perfectly fine (scroll, match all, etc..). I can provide details if you are still interested — animageofmine, Nov 02 '16 at 08:22
@ruhungry I stress tested the ES cluster on AWS and it changed the status to red, so I don't have the latency numbers for now. Filed a ticket and waiting for the cluster to be back to green. There might be a few milliseconds of difference for round trip time (RTT) between the native ES client and JEST, because there is some optimization that native client contains (information about nodes in the cluster). But AWS does not support transport protocol, so I don't know of any way to connect to ES. — animageofmine, Nov 03 '16 at 20:24

score 2 · Answer 1 · answered Oct 20 '16 at 05:37

2

Try to add .readTimeout to your client creation code:

factory.setHttpClientConfig(new HttpClientConfig.Builder(END_POINT)
            .multiThreaded(true)
            .readTimeout(60000)
            .build());

I was facing the same SocketTmeoutException: Read timed out.

Adding .readTimeout solved it for me. Hope this helps for you as well.

answered Oct 20 '16 at 05:37

Khurram Khan

51
7

1

Default readTimeout is set: `protected Integer readTimeout = 3000;` and multiplying it by 20 does not seem to be a good solution in production. – ruhungry Oct 20 '16 at 10:02
What 3000 means? 3 seconds? – Alin Feb 12 '17 at 19:15
Yes it takes millsecond as argument – Khurram Khan Feb 14 '17 at 04:15

AWS ElasticSearch Service version 2.3 does not work with Jest Client: java.net.SocketTimeoutException

1 Answers1