Are we able to retain all the persisted RDDs if we lose spark context

Question

Let me elaborate my question:

I am using a cluster which contains a master node and 3 worker node, my master

node has spark context available.

I have saved my RDD into the disk using storage level "DISK_ONLY".

When I run my spark script it will save some RDD to hard disk of any worker

node, now when my master machine goes down, which has spark context and as a

result it will also go down, thus all the DAG information lost.

Now I have to restart my master node so as to make spark context up and

running again.

now the question is - will I be able to retain all saved RDD back with this

bouncing (restarting master node and spark context daemon)? as everything is

restarted.

yes, seem for me i need to recompute back again and create new intermediate RDDs, i need to search what is the best way if need to reuse my saved ones, given the situation i explained — intellect_dp, Apr 07 '19 at 05:46
All about relationship master, driver. https://stackoverflow.com/questions/35815386/what-happens-when-spark-master-fails — thebluephantom, Apr 07 '19 at 07:02
Thanks for sharing the link @thebluephantom , i already visited this link before, but in my case master "SC" and driver coordinator exist in same node. — intellect_dp, Apr 07 '19 at 08:24
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/191406/discussion-between-intellect-dp-and-thebluephantom). — intellect_dp, Apr 07 '19 at 18:18

score 4 · Answer 1 · answered Apr 08 '19 at 05:04

I don't think currently there is a way to restore cached RDD after shutting down the Spark Context. The component that puts and gets RDD blocks is the BlockManager component of Spark. This, in turn, uses another component named BlockInfoManager to keep track of RDD block info. When a BlockManager shuts down in a worker node, it clears the resources that it was using. Among them is the BlockInfoManager, which has the HashMap containing the RDD block info. As this Map is also cleared in the process of cleaning up, when next time it is instantiated, there is no info of any RDD blocks being saved in that worker. Thus it will treat that block as uncomputed.

That I suspect is the case. – thebluephantom Apr 08 '19 at 07:52 — thebluephantom, Apr 08 '19 at 07:52

score 1 · Answer 2 · edited Apr 07 '19 at 13:59

1

According to @intellect_dp explanation, if you are using any cluster manager for example - Apache Mesos or Hadoop Yarn, then you need to specify which deploy mode you want to go with , "cluster mode" or "client mode",

Deploy mode distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.

edited Apr 07 '19 at 13:59

intellect_dp

169
1
12

answered Apr 07 '19 at 13:50

deepika patel

106
2

i am using yarn as the cluster management system, deploy cluster mode, but does that mean i need to use client mode to retain my RDDs – intellect_dp Apr 07 '19 at 14:05
1

My point is u need a solution where driver node and spark context they both are at different node, and keep SC node fault tolerant with help of yarn, in case SC node goes down – deepika patel Apr 07 '19 at 14:09
Not sure how question answered with this explanation. – thebluephantom Apr 07 '19 at 14:49
@thebluephantom - My explanation to the question, is inclining towards the ans where one need to keep SC node fault tolerant, so that next time if want to retain RDDs, SC's DAG will help us to retain all the RDDs – deepika patel Apr 07 '19 at 17:57
1

Others posts on SO point to it being different. I have to say - I am going to research this, and it is not an easy topic to understand. I am not sure the persist helps in this way, another topic for next weekend. Of course with Zookeeper you can have a standbyby Master. – thebluephantom Apr 07 '19 at 18:06
Yes, i admit this is not that easy, for sure it will include experiment and research – deepika patel Apr 07 '19 at 18:10

thebluephantom · Answer 3 · 2019-04-08T17:21:01.357

1

The short answer is NO. Best to failover your Master.

Alternatively or complimentary you could split up your jobs using a scheduler and use Spark bucketBy approach.

edited Apr 08 '19 at 17:21

answered Apr 08 '19 at 15:37

thebluephantom

16,458
8
40
83

Are we able to retain all the persisted RDDs if we lose spark context

3 Answers3