25

I don't quite understand spark.task.cpus parameter. It seems to me that a “task” corresponds to a “thread” or a "process", if you will, within the executor. Suppose that I set "spark.task.cpus" to 2.

  1. How can a thread utilize two CPUs simultaneously? Couldn't it require locks and cause synchronization problems?

  2. I'm looking at launchTask() function in deploy/executor/Executor.scala, and I don't see any notion of "number of cpus per task" here. So where/how does Spark eventually allocate more than one cpu to a task in the standalone mode?

smz
  • 391
  • 1
  • 3
  • 11

2 Answers2

31

To the best of my knowledge spark.task.cpus controls the parallelism of tasks in you cluster in the case where some particular tasks are known to have their own internal (custom) parallelism.

In more detail: We know that spark.cores.max defines how many threads (aka cores) your application needs. If you leave spark.task.cpus = 1 then you will have #spark.cores.max number of concurrent Spark tasks running at the same time.

You will only want to change spark.task.cpus if you know that your tasks are themselves parallelized (maybe each of your task spawns two threads, interacts with external tools, etc.) By setting spark.task.cpus accordingly, you become a good "citizen". Now if you have spark.cores.max=10 and spark.task.cpus=2 Spark will only create 10/2=5 concurrent tasks. Given that your tasks need (say) 2 threads internally the total number of executing threads will never be more than 10. This means that you never go above your initial contract (defined by spark.cores.max).

marios
  • 8,874
  • 3
  • 38
  • 62
  • 5
    I would add that it is used all over the source as `CPUS_PER_TASK` although it is not really enforced on runtime (more resource utilization hint than a hard limitation) – zero323 Apr 17 '16 at 14:55
  • do you have any example showing how a task could internally be parallelized? @zero323, CPUS_PER_TASK is used in a very high level scheduling stuff, it is never being passed to the executor. So, my question is "how do we make sure that a task gets spark.task.cpus cores that it requested?" – smz Apr 17 '16 at 20:06
  • 2
    The simplest way (not very useful): `rdd.mapPartitions(_.toVector.par.map(foo))`. And to answer your second question - we simply don't. It only says that if you declare four executor cores, and `spark.task.cpus` equal 2 then it can start only 2 tasks. But there is physical cores allocation or similar stuff. `core` in Spark is just a thread. – zero323 Apr 17 '16 at 20:34
  • @marios How to set this property spark.task.cpus ? Can I set in spark-submit command? – Kishore May 26 '18 at 05:44
  • @Kishore you can pass that via a --conf parameter in spark-submit, – marios May 27 '18 at 03:28
  • If I understand this correctly a single task on an executor can use any number of threads in parallell, that is not something spark decides? Let's say I run the sql.functions.from_json function on a column in a dataframe. Could that function then use all available cores to deserialize the dataframe even though it takes place in a single task on the executors? – Molotch Jul 22 '19 at 09:10
0

if I have set spark.task.cpus to 4 and spark.executor.cores to 4 -- this would mean I am spinning up one task per executor. Is this understanding correct?

  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 21 '23 at 22:39
  • This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stackoverflow.com/help/whats-reputation), you can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/34573285) – Andrew Jun 23 '23 at 05:36