0

I have 1 driver and 6 core instances with 16GB ram and 8 cores each. I am running spark-submit with below options:

spark-submit --driver-memory 4g \
    --executor-memory 6g \
    --num-executors 12 \
    --executor-cores 2 \
    --conf spark.driver.maxResultSize=0 \
    --conf spark.network.timeout=800 job.py

I am getting Java heap memory error multiple times, I think there is something wrong with the options can someone help me out with this.

Thanks

ernest_k
  • 44,416
  • 5
  • 53
  • 99
Raghav salotra
  • 820
  • 1
  • 11
  • 23

0 Answers0