Deserialization errors in Spark cluster although all objects implement Serializable

Question

I'm writing a java spark app, and I came across an invalid lambda deserialization error. Here is the relevant code:

protected void execute(JavaSparkContext javaSparkContext) throws Exception {
    ObjectMapper mapperWithNulls = MY_FACTORY.newObjectMapper();
    mapperWithNulls.setSerializationInclusion(Include.ALWAYS);
    SQLContext sqlContext = new SQLContext(javaSparkContext);

    JavaRDD<MyClass> myRdd = javaSparkContext.textFile(inputPath)
                                .map(s -> mapperWithNulls.readValue(s, MyClass.class));
}

This code throws the deserialization error, even though every class implements the serializable interface. However, if I move the ObjectMapper declaration out of method scope, like this:

final static ObjectMapper mapperWithNulls = MY_FACTORY.newObjectMapper();

protected void execute(JavaSparkContext javaSparkContext) throws Exception {
    mapperWithNulls.setSerializationInclusion(Include.ALWAYS);
    SQLContext sqlContext = new SQLContext(javaSparkContext);

    JavaRDD<MyClass> myRdd = javaSparkContext.textFile(inputPath)
                                .map(s -> mapperWithNulls.readValue(s, MyClass.class));
}

then the program runs fine. I would like to know why the first code example doesn't work, while the second one does. Thank you.

Edit: here is the stacktrace

java.io.IOException: unexpected exception type
    at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1582)
    at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1154)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1810)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1148)
    ... 27 more
Caused by: java.lang.IllegalArgumentException: Invalid lambda deserialization
    at com.mypackage.MyApp.$deserializeLambda$(MyApp.java:27)
    ... 37 more

The line in question is lombok's @CommonsLog