Apache Ignite's Unknown Exception

Question

"" "172.21.205.127" "test" "" "" "" "141" "org.apache.ignite.logger.java.JavaLogger" "error" "SEVERE" "04-04-2020 01:46:33:332" "309" "Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=5181477754, bytesSent=8063402035, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=null, finished=false, heartbeatTs=1585944947862, hashCode=1889540503, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#25]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=7451 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=37599296, resendCnt=0, rcvCnt=37267616, sentCnt=37599319, reserved=true, lastAck=37267616, nodeLeft=false, node=TcpDiscoveryNode [id=89223af4-b220-40b5-8576-0217b6baaeb8, consistentId=225589bc-ed9d-4239-908d-6194bbe4e55f, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.21.205.127, fda7:e6ee:2e09:0:1818:7b9:83b4:b1c4%enp0s31f6, fda7:e6ee:2e09:0:21ae:baed:76d2:ddf8%enp0s31f6, fda7:e6ee:2e09:0:59c0:6820:b96f:c121%enp0s31f6, fda7:e6ee:2e09:0:909e:8898:ac40:cf21%enp0s31f6, fda7:e6ee:2e09:0:ad5f:d0a5:d3:694f%enp0s31f6, fda7:e6ee:2e09:0:c5b8:368a:456a:b86b%enp0s31f6, fda7:e6ee:2e09:0:ccd5:da6e:7145:dbe7%enp0s31f6, fda7:e6ee:2e09:0:e5e6:74c5:5edc:3085%enp0s31f6, fdfe:9042:c53d:0:1818:7b9:83b4:b1c4%enp0s31f6, fdfe:9042:c53d:0:21ae:baed:76d2:ddf8%enp0s31f6, fdfe:9042:c53d:0:59c0:6820:b96f:c121%enp0s31f6, fdfe:9042:c53d:0:909e:8898:ac40:cf21%enp0s31f6, fdfe:9042:c53d:0:ad5f:d0a5:d3:694f%enp0s31f6, fdfe:9042:c53d:0:ccd5:da6e:7145:dbe7%enp0s31f6, fdfe:9042:c53d:0:e3af:2792:c94b:8c7d%enp0s31f6, fdfe:9042:c53d:0:e5e6:74c5:5edc:3085%enp0s31f6], sockAddrs=[/fda7:e6ee:2e09:0:59c0:6820:b96f:c121%enp0s31f6:47500, /fdfe:9042:c53d:0:59c0:6820:b96f:c121%enp0s31f6:47500, /fda7:e6ee:2e09:0:ccd5:da6e:7145:dbe7%enp0s31f6:47500, /fdfe:9042:c53d:0:e5e6:74c5:5edc:3085%enp0s31f6:47500, /fdfe:9042:c53d:0:21ae:baed:76d2:ddf8%enp0s31f6:47500, /fdfe:9042:c53d:0:e3af:2792:c94b:8c7d%enp0s31f6:47500, /fda7:e6ee:2e09:0:1818:7b9:83b4:b1c4%enp0s31f6:47500, /fdfe:9042:c53d:0:1818:7b9:83b4:b1c4%enp0s31f6:47500, /fda7:e6ee:2e09:0:ad5f:d0a5:d3:694f%enp0s31f6:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /fda7:e6ee:2e09:0:21ae:baed:76d2:ddf8%enp0s31f6:47500, /fdfe:9042:c53d:0:ad5f:d0a5:d3:694f%enp0s31f6:47500, /fdfe:9042:c53d:0:ccd5:da6e:7145:dbe7%enp0s31f6:47500, /fda7:e6ee:2e09:0:e5e6:74c5:5edc:3085%enp0s31f6:47500, /fda7:e6ee:2e09:0:c5b8:368a:456a:b86b%enp0s31f6:47500, /172.21.205.127:47500, /fda7:e6ee:2e09:0:909e:8898:ac40:cf21%enp0s31f6:47500, /fdfe:9042:c53d:0:909e:8898:ac40:cf21%enp0s31f6:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1585562622234, loc=false, ver=8.7.7#20191030-sha1:cfb0e3c4, isClient=false], connected=true, connectCnt=306, queueLimit=4096, reserveCnt=1081, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=37599296, resendCnt=0, rcvCnt=37267616, sentCnt=37599319, reserved=true, lastAck=37267616, nodeLeft=false, node=TcpDiscoveryNode [id=89223af4-b220-40b5-8576-0217b6baaeb8, consistentId=225589bc-ed9d-4239-908d-6194bbe4e55f, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.21.205.127, fda7:e6ee:2e09:0:1818:7b9:83b4:b1c4%enp0s31f6, fda7:e6ee:2e09:0:21ae:baed:76d2:ddf8%enp0s31f6, fda7:e6ee:2e09:0:59c0:6820:b96f:c121%enp0s31f6, fda7:e6ee:2e09:0:909e:8898:ac40:cf21%enp0s31f6, fda7:e6ee:2e09:0:ad5f:d0a5:d3:694f%enp0s31f6, fda7:e6ee:2e09:0:c5b8:368a:456a:b86b%enp0s31f6, fda7:e6ee:2e09:0:ccd5:da6e:7145:dbe7%enp0s31f6, fda7:e6ee:2e09:0:e5e6:74c5:5edc:3085%enp0s31f6, fdfe:9042:c53d:0:1818:7b9:83b4:b1c4%enp0s31f6, fdfe:9042:c53d:0:21ae:baed:76d2:ddf8%enp0s31f6, fdfe:9042:c53d:0:59c0:6820:b96f:c121%enp0s31f6, fdfe:9042:c53d:0:909e:8898:ac40:cf21%enp0s31f6, fdfe:9042:c53d:0:ad5f:d0a5:d3:694f%enp0s31f6, fdfe:9042:c53d:0:ccd5:da6e:7145:dbe7%enp0s31f6, fdfe:9042:c53d:0:e3af:2792:c94b:8c7d%enp0s31f6, fdfe:9042:c53d:0:e5e6:74c5:5edc:3085%enp0s31f6], sockAddrs=[/fda7:e6ee:2e09:0:59c0:6820:b96f:c121%enp0s31f6:47500, /fdfe:9042:c53d:0:59c0:6820:b96f:c121%enp0s31f6:47500, /fda7:e6ee:2e09:0:ccd5:da6e:7145:dbe7%enp0s31f6:47500, /fdfe:9042:c53d:0:e5e6:74c5:5edc:3085%enp0s31f6:47500, /fdfe:9042:c53d:0:21ae:baed:76d2:ddf8%enp0s31f6:47500, /fdfe:9042:c53d:0:e3af:2792:c94b:8c7d%enp0s31f6:47500, /fda7:e6ee:2e09:0:1818:7b9:83b4:b1c4%enp0s31f6:47500, /fdfe:9042:c53d:0:1818:7b9:83b4:b1c4%enp0s31f6:47500, /fda7:e6ee:2e09:0:ad5f:d0a5:d3:694f%enp0s31f6:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /fda7:e6ee:2e09:0:21ae:baed:76d2:ddf8%enp0s31f6:47500, /fdfe:9042:c53d:0:ad5f:d0a5:d3:694f%enp0s31f6:47500, /fdfe:9042:c53d:0:ccd5:da6e:7145:dbe7%enp0s31f6:47500, /fda7:e6ee:2e09:0:e5e6:74c5:5edc:3085%enp0s31f6:47500, /fda7:e6ee:2e09:0:c5b8:368a:456a:b86b%enp0s31f6:47500, /172.21.205.127:47500, /fda7:e6ee:2e09:0:909e:8898:ac40:cf21%enp0s31f6:47500, /fdfe:9042:c53d:0:909e:8898:ac40:cf21%enp0s31f6:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1585562622234, loc=false, ver=8.7.7#20191030-sha1:cfb0e3c4, isClient=false], connected=true, connectCnt=306, queueLimit=4096, reserveCnt=1081, pairedConnections=false], super=GridNioSessionImpl [locAddr=/0:0:0:0:0:0:0:1:47101, rmtAddr=/0:0:0:0:0:0:0:1:51694, createTime=1585944379990, closeTime=0, bytesSent=88972787, bytesRcvd=85003001, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1585944379990, lastSndTime=1585944947862, lastRcvTime=1585944947862, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@35a28147, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=true]]]" "java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
    at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1683)
    at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1340)
    at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2414)
    at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2178)
    at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1819)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)

I am receiving this exception sometimes and I could see this in my Ignite's client node logs. Can someone please tell me this?

What is the significance of this issue? When this will occur? What do we need to do to fix if it causes any problem.

Does this answer your question? [When is "java.io.IOException:Connection reset by peer" thrown?](https://stackoverflow.com/questions/8658118/when-is-java-io-ioexceptionconnection-reset-by-peer-thrown) — dawis11, Apr 13 '20 at 12:27

score 0 · Answer 1 · answered Apr 13 '20 at 12:32

0

It is possible that your network termiates long-running connections with no activity on them. This is bad for Apache Ignite cluster since nodes don't know whether their connections to each other work or not.

answered Apr 13 '20 at 12:32

alamar

18,729
4
64
97

So you are saying this could be our network environment config problem? We do have our office's network proxy config which our running client app uses. – RSSAH Apr 13 '20 at 12:45
Can you please suggest what we could regarding our network for this? I want to know what you meant by long-running connections? You mean ignite node-node connection will never terminate till the application is up and running? – RSSAH Apr 13 '20 at 13:09
Yes, they will not terminate until node is down. – alamar Apr 13 '20 at 13:27
Hi alamar. when I checked top command, it's saying 100% java CPU used for a long time. Still not decreasing.. happening for both java nodes.. client(app server node) and server nodes.. both running on same machine.. it's flickering between 90% to 100% CPU.. but memory percentage is under 30%. even in logs i'm getting jvm pause alerts.. So for both processes i ran jmap.. do you want to see those files? – RSSAH Apr 14 '20 at 06:55
please share your mail address. So that I'll send hprofs and thread dump txt files – RSSAH Apr 14 '20 at 07:27

Apache Ignite's Unknown Exception

1 Answers1