Our spring boot app is indexing data in ES cluster using transport client [using ElasticsearchRepository ]. Issue is that after some index call indexing just hangs the.the SAVE method on DAO just hangs and After some time it gives us "Nonodefoundexception"
Here’s our stack
- ES 1.7.3
- Spring Boot: 1.3.2.RELEASE
- Java: 1.8
- spring-boot-starter-data-elasticsearch : 1.3.2.RELEASE
Using transport client:
ES setup: - there are three nodes. all are client and master and data
Environment: Our dev is in AWS and we do not see any issue in Dev even though all ES setup is same.
**org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[NodeName-apsrp5470][TzlZ3u8aTK23YFr14R085g][apsrp5470][inet[/10.222.146.127:9300]]{master=true}, [NodeName-apsrp5434][K4r0wujuThqwj1oTU9lEHg][apsrp5434][inet[/10.222.146.132:9300]]{master=true}, [NodeName-apsrp5435][YBPk8E4TR6u7LFLxfaVWaw][apsrp5435][inet[/10.222.146.139:9300]]{master=true}]
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:242) ~[elasticsearch-1.5.2.jar!/:na]
at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:78) ~[elasticsearch-1.5.2.jar!/:na]
at org.elasticsearch.transport.TransportService$Adapter$3.run(TransportService.java:468) ~[elasticsearch-1.5.2.jar!/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [NodeName-apsrp5470][inet[/10.222.146.127:9300]][indices:data/write/index] disconnected**
Thread Dumps :-
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bd824f48> (a org.elasticsearch.common.util.concurrent.BaseFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:279)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:118)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:45)
at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.index(ElasticsearchTemplate.java:519)