I tried copying the logging trait from the spark to my project. Previously I was using print instead of log. The below code was working fine with print statement. But after adding the log, I am getting task not serializable error. This is due to the log is referred by outer class and the closure is trying to serialize the entire class object.
The issue can be solved by making the whole class as serializable. But my questions are
1) Is it worth to serialize the whole class? what are the pros/cons of it?
2) I tried adding another object and use that logger inside the parserFunction it works but Is there any other efficient way to get the log inside the function ?
class SampleClass(interval: String) extends Logging {
// other stuffs
def getParserFunction():
(CassandraRow) => List[String] = {
(row: CassandraRow) =>
try {
// doSomeThing
}
catch {
case NonFatal(ex) => {
logWarning("invalid data:")
}
}
}
}