1

I am planning a new spark cluster. My question is do I need install spark worker node on all data node? for example, if I have 50 data nodes and I only install 10 spark worker node, is it reasonable? Thanks!

Wang Nan
  • 21
  • 4
  • 5
    It depends on Spark mode you want to use, For standalone you need to install on all nodes, for yarn you just need to install on one node where you will be launching your spark jobs. – vgunnu May 11 '16 at 21:51
  • Please check this link: http://stackoverflow.com/questions/24909958/spark-on-yarn-concept-understanding/38598830#38598830 – mrsrinivas Jul 26 '16 at 20:04

1 Answers1

0

Yes you need to install spark worker on all nodes because which node will be used you cannot predict by default. Please find this link for multinode spark installation

KP-
  • 32
  • 3