elasticsearch out of memory error

Question

We are using elasticsearch 0.90.0 and java version "1.7.0_25". We migrate data from oracle DB to Hadoop through a executable jar kept at DB server.after 15-20 mins of successful running, we get following exception

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:597)
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:51)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
    at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:240)
    at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
    at org.elasticsearch.transport.TransportService.doStart(TransportService.java:90)
    at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:179)
    at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:119)

No exception is caught either in namenode/datanode or elasticsearch logs. This error is caught at DB Server but I think it is related to elastic search.

How many indices do you have? How much memory is allocated to Elastic Search? Is this a recent occurrence (assume it is since you are running an old version of ES). You may have crossed a point where you either need to delete old indices or increase the memory allocated to ES. — Alcanzar, Aug 07 '14 at 14:08

score 0 · Answer 1 · edited May 23 '17 at 12:19

0

Looks like you're creating too many Netty clients is my guess which is in turn eating up all your threads. Perhaps wrap your Netty client pool with a service that inject? See Helter Skelter's comment on this answer https://stackoverflow.com/a/5253186/266531

edited May 23 '17 at 12:19

Community

1
1

answered Jan 07 '15 at 22:28

Kirby

15,127
10
89
104

elasticsearch out of memory error

1 Answers1