2

I am new to spark. When I try to run spark-submit in client mode with 3 executors , I expect 3 java processes (since there are 3 executors ) to show up when I execute ps -ef 

$SPARK_HOME/bin/spark-submit --num-executors 3 --class AverageCalculation --master local[1] /home/customer/SimpleETL/target/SimpleETL-0.1.jar hdfs://node1:9000/home/customer/SimpleETL/standard_input.csv

But, I dont see 3 java processes. My undrstanding is that each executor process is a java process. Please advise. Thanks.

SCB
  • 5,821
  • 1
  • 34
  • 43
osr
  • 21
  • 1
  • 3

4 Answers4

6

Because you use local mode (--master local[1]) executor settings are not applicable. In this case, spark starts only a single JVM to emulate all components, and allocates number of threads specified in local definition (1) as executor threads.

In other modes, exectuors are separate JVM instances.

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115
2

Each executors are a java process. Each executors comprises a jvm.

jps

Number of java process is same as the number of executors. If the executors are distributed across the worker nodes. Need to check the process the corresponding worker nodes. We can get the information about executors and where it has been launched from spark history server web UI.

Radhakrishnan Rk
  • 553
  • 3
  • 13
0

In Spark, there are master nodes and worker nodes. Executors run on worker nodes in their own java processes.

In your spark-submit you can add --deploy-mode cluster and see that executors are running on worker nodes in their own JVM instances.

You can check this answer for detailed workflow of Apache Spark.

Shrinivas Deshmukh
  • 667
  • 5
  • 12
0
/home/spark/spark-2.2.1-bin-hadoop2.7/bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --num-executors 1000 \
    --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 \
    --queue default /home/spark/spark-2.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.1.jar

~ I executed the command above. and checked ps -ef | grep java. But I dont see a lot of java processes . Any easy way to identify the executors ?

osr
  • 21
  • 1
  • 3