I have been trying to run Spark on Hadoop to run applications, but everything seems fine and I get an application to success but when O see the application tracking UI of Spark, it only show to nodes to be having the containers of a 4 node cluster (inclusive of master). I am not able to configure each node to have one cluster.
Kindly help with the solution
The application UI
The Spark UI
The Spark conf file
When Spark submit the Job to the YARN resource manager, it draws a logical and physical execution plan based on data size, partition, data locality and accordingly the number of executors is planned and it all happens automatically. You can still configure # of executor required however whether to run them in a single node or in the different nodes in the cluster or in the specific node that depends on data locality and kind of job you have submitted. You can not instruct YARN to run all the executors in all the nodes in the cluster but if you have a very large data set and complex transformation, it will automatically use all the cluster in the node.
