Timeout Exception in Apache-Spark during program Execution

Question

I am running a Bash Script in MAC. This script calls a spark method written in Scala language for a large number of times. I am currently trying to call this spark method for 100,000 times using a for loop.

The code exits with the following exception after running a small number of iterations, around 3000 iterations.

org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval
    at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:518)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:547)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:547)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:547)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1877)
    at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:547)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

Exception in thread "dag-scheduler-event-loop" 16/11/22 13:37:32 WARN NioEventLoop: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
    at io.netty.util.internal.MpscLinkedQueue.offer(MpscLinkedQueue.java:126)
    at io.netty.util.internal.MpscLinkedQueue.add(MpscLinkedQueue.java:221)
    at io.netty.util.concurrent.SingleThreadEventExecutor.fetchFromScheduledTaskQueue(SingleThreadEventExecutor.java:259)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:346)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:745)
java.lang.OutOfMemoryError: Java heap space
    at java.util.regex.Pattern.compile(Pattern.java:1047)
    at java.lang.String.replace(String.java:2180)
    at org.apache.spark.util.Utils$.getFormattedClassName(Utils.scala:1728)
    at org.apache.spark.storage.RDDInfo$$anonfun$1.apply(RDDInfo.scala:57)
    at org.apache.spark.storage.RDDInfo$$anonfun$1.apply(RDDInfo.scala:57)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.storage.RDDInfo$.fromRdd(RDDInfo.scala:57)
    at org.apache.spark.scheduler.StageInfo$$anonfun$1.apply(StageInfo.scala:87)

Can someone help please, is this error being caused because of a large number of calls to spark method?

it also show java out of space exception so can you once increase the memory and try it again — Sandeep Purohit, Nov 22 '16 at 12:05
Are you prsisting data? With cache or something like that? Are you using Dataframe or RDD api? — Thiago Baldim, Nov 22 '16 at 12:10
Can you add the snipped of the code? Maybe it can be from your method. — Thiago Baldim, Nov 22 '16 at 15:01
@RamPrasadG I am still running it. I have very large dataset. Once it successfully completes without error I shall accept the answer. Last time the error occurred after a day of execution. thanks for the help — Yasir Arfat, Nov 23 '16 at 17:56

Ram Ghadiyaram · Accepted Answer · 2016-11-22T19:45:32.347

20

Its RpcTimeoutException .. so spark.network.timeout (spark.rpc.askTimeout) could be tuned with larger-than-default values in order to handle complex workload. You can start with these values and adjust accordingly to your workloads. Please see latest

spark.network.timeout 120s Default timeout for all network interactions. This config will be used in place of spark.core.connection.ack.wait.timeout, spark.storage.blockManagerSlaveTimeoutMs, spark.shuffle.io.connectionTimeout, spark.rpc.askTimeout or spark.rpc.lookupTimeout if they are not configured.

Also consider increasing executor memory i.e spark.executor.memory and most imp thing is review your code, to check whether that is candidate for further optimization.

Solution : value 600 is based on requirement

set by SparkConf: conf.set("spark.network.timeout", "600s")
set by spark-defaults.conf: spark.network.timeout 600s
set when calling spark-submit: --conf spark.network.timeout=600s

edited Nov 22 '16 at 19:45

answered Nov 22 '16 at 18:09

Ram Ghadiyaram

28,239
13
95
121

What would be the disadvantages of keeping a high value ? . Does it cause any side effects ? – Amogh Huilgol Jan 11 '19 at 15:19
@AmoghHuilgol : I haven't checked that! but it is clear that large workload needs bit high value. you can try on your own and post as separate question and answer for this. – Ram Ghadiyaram Jan 11 '19 at 17:44
@AmoghHuilgol :Pls give feedback if you like the answer here :-) – Ram Ghadiyaram Jan 11 '19 at 17:46

score 5 · Answer 2 · edited Nov 29 '17 at 11:19

5

The above stack trace is also shown java heap space its OOM error so once try to increase the memory and run it and regarding timeout its rpc timeout so you can set spark.network.timeout with timeout value according to your need...

edited Nov 29 '17 at 11:19

user3190018

890
13
26

answered Nov 22 '16 at 12:09

Sandeep Purohit

3,652
18
22

score 1 · Answer 3 · edited Aug 24 '17 at 18:21

1

pls increase the executer memory so that OOM will go away else make chnage in code so that your RDD wont have big memory foot print.

--executer-memory = 3G

edited Aug 24 '17 at 18:21

Ram Ghadiyaram

28,239
13
95
121

answered May 24 '17 at 16:49

Prem S

217
3
8

score 1 · Answer 4 · edited Nov 29 '17 at 09:46

1

Just increase the spark.executor.heartbeatInterval to 20s, the error says that.

edited Nov 29 '17 at 09:46

Ram Ghadiyaram

28,239
13
95
121

answered Oct 12 '17 at 07:39

Luckylukee

575
2
9
27

score 1 · Answer 5 · answered Jan 22 '20 at 22:31

1

You are seeing this issue due to the executor memory. Try increasing the memory to (x 2) so the containers don't time out while waiting on the remaining containers.

answered Jan 22 '20 at 22:31

akl

46
4

score 0 · Answer 6 · answered Dec 31 '21 at 19:37

For posterity: I was getting similar errors, but changing memory/timeout settings was not helping at all.

In my case the problem was that somebody was calling socket.setdefaulttimeout in a library function that I was calling before creating the Spark session. setdefaulttimeout affected all new sockets created after that point, including the socket that Spark used to communicate with YARN, so that connection would time out unexpectedly.

Needless to say, don't do this.

Timeout Exception in Apache-Spark during program Execution

6 Answers6

Solution : value 600 is based on requirement

Linked