3

I'm running Spark 2.0 in stand-alone mode, and I'm the only one submitting jobs in my cluster.

Suppose I have an RDD with 100 partitions and only 10 partitions in total would fit in memory at a time.

Let's also assume that allotted execution memory is enough and will not interfere with storage memory.

Suppose I iterate over the data in that RDD.

rdd.persist()  // MEMORY_ONLY

for (_ <- 0 until 10) {
  rdd.map(...).reduce(...)
}

rdd.unpersist()

For each iteration, will the first 10 partitions that are persisted always be in memory until rdd.unpersist()?

Russell
  • 3,975
  • 7
  • 37
  • 47
  • See also https://stackoverflow.com/questions/69597745/when-was-automatic-spark-rdd-partition-cache-eviction-implemented – samthebest Jul 28 '22 at 14:10

2 Answers2

5

For now what I know Spark is using LRU (Less Recently Used) eviction strategy for RDD partitions as a default. They are working on adding new strategies. https://issues.apache.org/jira/browse/SPARK-14289

This strategy remove an element which is less recently used The last used timestamp is updated when an element is put into the cache or an element is retrieved from the cache.

I suppose you will always have 10 partition in your memory, but which are stored in memory and which will get evicted depends on their use. According Apache FAQ:

Likewise, cached datasets that do not fit in memory are either spilled to disk or recomputed on the fly when needed, as determined by the RDD's storage level.

Thus, it depends on your configuration if other partitions are spilled to disk or recomputed on the fly. Recomputation is the default, which is not always most efficient option. You can set a dataset's storage level to MEMORY_AND_DISK to be able to avoid this.

bonnal-enzo
  • 1,165
  • 9
  • 19
Stefan Repcek
  • 2,553
  • 4
  • 21
  • 29
  • Thanks. From what I understand and from the JIRA link you included, LRU mainly concerns different RDDs being cached. Here I'd like to know what happens to a single RDD whose data partitions cannot be completely included in memory. – Russell Mar 08 '17 at 01:20
  • according what I understand, the same technique apply regardles if you have single RDD or multiple RDDs. – Stefan Repcek Mar 08 '17 at 09:21
  • 1
    The comment in the JIRA link bothered me. "The default RDD eviction strategy is LRU (with an additional rule that do not replacing another block that belongs to the same RDD like current creating partition)." I dug around a bit more, and I believe partitions are not evicted to make way for other partitions in the same RDD. Thanks for the pointer though! – Russell Mar 08 '17 at 17:48
3

I think I found the answer, so I'm going to answer my own question.

The eviction policy seems to be in the MemoryStore class. Here's the source code.

It seems that entries are not evicted to make place for entries in the same RDD.

Russell
  • 3,975
  • 7
  • 37
  • 47
  • do u know why "it would require replacing another block from the same RDD (which leads to a wasteful cyclic replacement pattern for RDDs that don't fit into memory that we want to avoid)."? – chenzhongpu Jan 17 '19 at 04:36