Apache Spark Configuration in Scala

Asked Jul 05 '19 at 14:19

Active Jul 05 '19 at 14:19

Viewed 33 times

I am using Apache Spark in my Scala application. The problem is that the application is using all CPU cores permanently(with avg of %95). I even tried to set spark.cores.max to 1 but it did not seem to help. My application is running in local (has 4 CPU cores). What might be the problem? Here are the versions and the code I used for creating Spark Configuration.

  val sparkConf = new SparkConf()
   .setAppName("MyApp")
   .setMaster("local[4]")
   .set("spark.cores.max", "1")

scala version: 2.11.12
spark version: 2.3.0

asked Jul 05 '19 at 14:19

Onur Arı

2

`local[4]` means use **4** CPUs, maybe `local[1]` is what you want? – Luis Miguel Mejía Suárez Jul 05 '19 at 14:24
I thought local[4] meant, Spark can use 4 different Threads. Then, I wanted all 4 threads to work on a single CPU core. – Onur Arı Jul 05 '19 at 14:25
By the way making it local[1] did not change anything. – Onur Arı Jul 05 '19 at 14:30
Maybe the excessive usage comes from GC? or other **Spark** programs running concurrently? Or maybe the value is being overwritten somehwere else? To be honest, it is hard to tell without more information, but `master[1]` SHOULD just use one CPU and it should be the option with highest priority. – Luis Miguel Mejía Suárez Jul 05 '19 at 14:36

Apache Spark Configuration in Scala

0 Answers0