I'm using Spark Streaming to deal with message delivered by Kafka. and now I came across with a problem.
There are several executors set on different machines to process those tasks, but there's only one executor, or to be specific, only executors on one machine is actually working while others remain free. Now tasks are heavily queued and I got oom alerts often.
Here is my config:
—driver-cores 1 —driver-memory 512m —executor-memory 512m
—conf spark.memory.useLegacyMode=true
—conf spark.eventLog.enabled=true
—conf spark.driver.extraJavaOptions='-Xmx512m -Xmx512m'
—conf spark.executor.extraJavaOptions='-XX:NewRatio=4 -XX:PermSize=100M'
—conf spark.dynamicAllocation.enabled=true
—conf spark.shuffle.service.enabled=true
—conf spark.dynamicAllocation.minExecutors=1
—conf spark.dynamicAllocation.maxExecutors=5
—conf spark.dynamicAllocation.initialExecutors=3
—conf spark.dynamicAllocation.executorIdleTimeout=60s
Screenshot:
