Passing Functions to Spark: What is the risk of referencing the whole object?

Question

According to Passing Functions to Spark,it claims:

accessing fields of the outer object will reference the whole object; To avoid this issue ...

I am considering that what is the risk of flowing code:

class MyClass {
  val field = "Hello"
  def doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(x => field + x) }
}

references all of this would do any harm ?

Are you just asking what the negative effects of this are? – Justin Pihony Mar 04 '15 at 03:05 — Justin Pihony, Mar 04 '15 at 03:05
yes. What are the negative effects ? – chenzhongpu Mar 04 '15 at 03:42 — chenzhongpu, Mar 04 '15 at 03:42

score 7 · Answer 1 · edited May 23 '17 at 12:24

This will cause Spark to serialize your whole object and send it to each of the executors. If some of the fields of your object contain big amounts of data, it might be slow. Also it might cause task not serializable exception if your object is not serializable

Here's an example of the guy with this problem: Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

Passing Functions to Spark: What is the risk of referencing the whole object?

1 Answers1

Linked