1

I have following spark configuration :

1 Master and 2 Workers

  • Each worker has 88 Cores , hence total no. of cores 176
  • Each worker has 502 GB memory , so total memory available is 1004 GB

now I want to run 40 executors so that all the cores will not be used.

I am running below command for same.

./spark-submit --class com.sample.Transformation --conf spark.sql.shuffle.partitions=5001 --num-executors=40 --executor-cores=1 --executor-memory=5G --master spark://10.180.181.41:7077 "/MyProject/Transformation-0.0.1-SNAPSHOT.jar" > /MyProject/logs/logs12.txt

Here i have sepcified --num-executors=40 still 176 executors has been created.

In same command When I changed --executor-cores=3it created 176/3 = 58 executors and 174 cores are used.

So it seems --num-executors value is not being considered in the command.

I want to understand why such behaviour of command and what can be resolution for same.

EDIT:

I am not using standalone mode here.

Raj
  • 707
  • 6
  • 23
  • @user8371915 I will try the formulae given in the referenced question, however that question is for standalone mode and I am not using standalone mode in my case. – Raj Aug 24 '18 at 04:01
  • @user8371915 Thank for pointing it out, it worked. – Raj Aug 24 '18 at 05:20
  • I am glad to hear that. And at least the code in the question, uses standalone mode. `spark://...` is standalone master URL. – Alper t. Turker Aug 24 '18 at 10:36

0 Answers0