Getting this error out of a sudden java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED

Question

I'm using httpcore-nio-4.4.5.jar. I'm using the elasticsearch RestHighLevelClient to interact with our elasticsearch servers. This all works fine except for some time we get I/O reactor stopped error out of a sudden.

Everything seems fine on ES side. No strange behaviour.

That's how i'm initializing my ES client.

public synchronized RestHighLevelClient getHighLevelClient() throws ManagerException {
        if (highLevelClient != null) {
            return highLevelClient;
        }

        Map<String, Integer> map = getEsServers(esAddresses);

        HttpHost[] hosts = new HttpHost[map.size()];

        int i = 0;

        for (Map.Entry<String, Integer> entry : map.entrySet()) {
            hosts[i++] = new HttpHost(entry.getKey(), entry.getValue(), "http");
            LOGGER.info(entry.getKey() + " " + entry.getValue());
        }

        RestClientBuilder restClientBuilder = RestClient.builder(hosts);
        highLevelClient = customizeHttpClient(restClientBuilder);
        return highLevelClient;
    }

public RestHighLevelClient customizeHttpClient(RestClientBuilder restClientBuilder) {
        Header[] defaultHeaders = new Header[2];
        defaultHeaders[0] = new BasicHeader("Authorization", "Basic YTph");
        defaultHeaders[1] = new BasicHeader("Accept", "application/json");

        restClientBuilder.setDefaultHeaders(defaultHeaders);

        restClientBuilder.setMaxRetryTimeoutMillis(MAX_RETRY_TIMEOUT_MILLIS);

        restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                .setConnectTimeout(CONNECT_TIMEOUT_MILLIS)
                .setSocketTimeout(SOCKET_TIMEOUT_MILLIS)
                .setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT_MILLIS));

        restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                .setMaxConnPerRoute(MAX_CONN_PER_ROUTE)
                .setMaxConnTotal(MAX_CONN_TOTAL));

        return new RestHighLevelClient(restClientBuilder);
    }

So basically first I get the following stacktrace

java.lang.IllegalStateException: I/O reactor has been shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34) 
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:224) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:434) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:276) 
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:266) 
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:363) 
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:125) 
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515) 
        at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400)

and after that no Timeout just the following exception continuously until i restart my servers.

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
    at org.apache.http.util.Asserts.check(Asserts.java:46)  
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)  
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515)  
    at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400)  ```

Possible duplicate of [java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED](https://stackoverflow.com/questions/35350376/java-lang-illegalstateexception-request-cannot-be-executed-i-o-reactor-status) — Kalpesh Soni, May 28 '19 at 22:39
@KalpeshSoni I tried that already, Couldnt find anything there. — Harshit Gupta, May 30 '19 at 19:41

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

I was facing the same issue, and it turned out I was not closing the HighLevelRestClient.

Refer to HenningAndersen answer at https://discuss.elastic.co/t/request-cannot-be-executed-i-o-reactor-status-stopped/195438/4:

This sounds similar to https://github.com/elastic/elasticsearch/issues/45115 141. The problem in that issue is caused by throwing an exception in an onFailure method in the client application code. This causes the connection to be closed and subsequent requests will fail with the error reported here.

I believe this also affects the high level client if using any of the methods reporting response/failure back through an ActionListener (I think all of them have suffix Async in their method names).

The workaround is to ensure that no exceptions are thrown out of your onFailure methods.

score 0 · Answer 2 · answered Mar 02 '22 at 01:44

0

Change the condition of getHighLevelClient() to if (highLevelClient != null && highLevelClient.getLowLevelClient().isRunning())

which means if the low level client (RestClient) is not running, you should renew the client, that works for me.

answered Mar 02 '22 at 01:44

halfdark

50
8

7.10.0 version of `org.elasticsearch:elasticsearch` related dependencies provide `isRunning()` method – halfdark Mar 02 '22 at 01:49

Getting this error out of a sudden java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED

2 Answers2