2
public static void main(String[] args) {
        List<String> inputData = new ArrayList<>();
        inputData.add("WARN: Tuesday 4 September 0405");
        inputData.add("ERROR: Tuesday 4 September 0408");
        inputData.add("FATAL: Wednesday 5 September 1632");
        inputData.add("ERROR: Friday 7 September 1854");
        inputData.add("WARN: Saturday 8 September 1942");

        Logger.getLogger("org.apache").setLevel(Level.WARN);

        SparkConf conf = new SparkConf().setAppName("startingSpark").setMaster("local[*]");
        JavaSparkContext sc = new JavaSparkContext(conf);

        sc.parallelize(inputData)
        .flatMap(value -> Arrays.asList(value.split(" ")).iterator())
        .filter(word -> word.length() > 1)
        // Here comes the exception
        .foreach(System.out::println);
        sc.close();
    }

foreach(s -> System.out.println(s)); --> Works fine

foreach( System.out::println ); --> Exception in thread "main" org.apache.spark.SparkException: Task not serializable

both are same, latter is using method reference feature from java 8 onwards, but it produces Exception in thread "main" org.apache.spark.SparkException: Task not serializable, why is it so?

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
Susmit Sarkar
  • 57
  • 5
  • I cannot get a full landscape of your question by the content above, but it seems like this [post](https://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou) will offer you some help. – Lebecca Feb 22 '20 at 18:39
  • which versions of scala and spark are you using? – Levi Ramsey Feb 22 '20 at 19:03
  • @LeviRamsey Scala is not used here I am using Java APIs version 8 and Spark version is 2.0.0 – Susmit Sarkar Feb 23 '20 at 11:53
  • @Lebecca, thank you but I have already checked the post before posting here, here the error is a little different. I have updated my code snippet for better understanding. Hope it helps!! – Susmit Sarkar Feb 23 '20 at 12:00
  • `SparkConf ` from which package, I import spark core only get `JavaSparkContext` exist. – Lebecca Feb 24 '20 at 06:09

0 Answers0