1

I think I'm familiar with Spark Cluster Mode components which described here and one of its major disadvantage - shuffling. hence, IMHO a best practice would be to have one executor for each worker node, having more than that will unnecessarily increase the shuffling (between the executors).

What am I missing here, in which cases I would prefer to have more then one executor in each worker?

Maor Aharon
  • 312
  • 3
  • 14

1 Answers1

-1

In which case would we like to have more than one executor in each worker?

Whenever possible: if jobs require less resources for one executor than what a worker node has, then spark should try to start other executors on the same worker to use all its available resources.

But that's the role of spark, not our call. When deploying spark apps, it is up to spark to decide how many executors (jvm process) are started on each worker node (machine). And it depends on the executor resources (core and memory) required by the spark jobs (the spark.executor.* configs). We often don't know what resources are available per worker. A cluster is usually shared by multiple apps/people. So we configure executor number and required resources and let spark decide to run them on the same worker or not.

Now, your question is maybe "should we have less executors with lots of cores and memory, or distribute it in several little executors?"

Having less but bigger executors reduce shuffling, clearly. But there are several reasons to also prefer distribution:

  • It is easier to start little executors
    • Having big executor means the cluster need all required resources on one worker
    • it is specially useful when using dynamic allocation, that will start and kill executor in function of runtime usage
  • Several little executors improve resilience: if our code is unstable and might sometime kill the executor, everything is lost and restarted.
  • I met a case where the code used in the executor wasn't thread safe. That's a bad thing, but it wasn't done on purpose. So until, or instead :\ , this is fixed, we distributed the code on many 1-core executors.
Juh_
  • 14,628
  • 8
  • 59
  • 92
  • Regarding your first part, obviously each executor can have one or more tasks, I didn't ask if we should limit the tasks per executor rather than why to increase the number of executors per worker node. Furthermore, it's true that Spark has default values, I'm not asking if they are good or bad, they are probably good for 90% of the cases, I'm asking in which cases we would like to overwrite them and have more than 1 executor per worker node. – Maor Aharon Oct 13 '21 at 17:12
  • Regarding your second part, it makes sense. I wounder if resilience is the only reason or having more than one executor per worker node could somehow boost the performance in some cases. – Maor Aharon Oct 13 '21 at 17:12
  • 1
    I think you're still confusing stuff: I didn't speak about task. Worker are machine/node, basically. For each spark app (i.e. program) started on the cluster, several executors are started (spark.executor.instances). Those are jvm instance started on some worker. And spark is the one to choose where. Then, a spark app will run jobs, one for each spark action (count, collect, write). And each job will be decomposed in tasks that are executed by the executor. – Juh_ Oct 14 '21 at 07:46
  • @Juh_ I gave up here, lost in translation. Can happen. – thebluephantom Oct 14 '21 at 09:17
  • :( hopefully it will be useful to someone... – Juh_ Oct 14 '21 at 11:48
  • I did not mean you to be clear – thebluephantom Oct 14 '21 at 13:17