I have a doubt regarding spark-submit for which I am not getting a clear solution. Running spark-submit in local machine with master as local[] will run it in local mode is what I am clear about but what if I run the same master as local[] in the gateway node or in the master node itself. Is my application consumes the entire cluster or is still running in local mode in that particular machine. Thanks
Asked
Active
Viewed 1,152 times
2 Answers
1
what if I run the same master as local[] in the gateway node or in the master node itself. Is my application consumes the entire cluster or is still running in local mode in that particular machine.
Answer : local mode in that particular machine ( gateway node or in the master node). since you are not using yarn as master. just try a sample test on this to prove the point.
For more details see my answer here
another way to see that is spark ui.. in environment tab it will display the details about where it is running... see the screen shot.
Ram Ghadiyaram
- 28,239
- 13
- 95
- 121
-
hi was it useful ? if you are okay please care to [accept answer as owner](https://meta.stackexchange.com/a/5235) – Ram Ghadiyaram Jul 03 '19 at 06:56
-
was it useful ? – Ram Ghadiyaram Jul 12 '19 at 15:42
0
In that case it will use the resources of gateway node or the master node. Cluster comes to role only when any of the cluster manager (E.g. YARN, Mesos etc) is mentioned in master.
vijayraj34
- 2,135
- 26
- 27