5

Could anyone give me a more precise description of this Spark parameter and how it effects program execution? I cannot tell exactly what this parameter does "under the hood" from the documentation.

cnnrznn
  • 435
  • 1
  • 4
  • 11

1 Answers1

3

The parameter influences the memory limit for Python workers. If the RSS of a Python worker process is larger than the memory limit, then it will spill data from memory to disk, which will reduce the memory utilization but is generally an expensive operation.

Note that this value applies per Python worker, and there will be multiple workers per executor.

If you want to take a look under the hood, then look at the python/pyspark directory in the Spark source tree, e.g. the ExternalMerger implementation: https://github.com/apache/spark/blob/41afa16500e682475eaa80e31c0434b7ab66abcb/python/pyspark/shuffle.py#L280

user2303197
  • 1,271
  • 7
  • 10
  • Follow up question: What does RSS stand for? Also, I though Spark used the term "worker" to describe each physical node the cluster is running on; how is it defined in this case? From your description, I take it to be equivalent to a spark "task." – cnnrznn Jul 22 '15 at 16:58
  • 1
    RSS is the resident set size (see here: http://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management). In the case above, a Python worker refers to a Python process executing a task in Spark (e.g. a chain of `map` functions on an RDD partition). – user2303197 Jul 22 '15 at 19:22