0

When i run my code from prod, Am not getting any task not serializable issue while when am trying to call from unit test case. am getting task not serializable from below code. Not getting what is the issue and why this strange behavior. Can anyone help on that or any better serialazable solution to get latest row from hive table ?

   val distinctBy = Window.partitionBy("id").orderBy(desc("updated_at"));
    val uniqueSellerDf = enrichedDf.withColumn("rank", rank().over(distinctBy))//row_number().over(distinctBy))     ### both rank function and row_number function have the same issue
      .where($"rank" === 1).drop("rank")

uniqueSellerDf.show()   //#### Getting task not serializable issue from show action cmd
Deepak Singh
  • 9
  • 2

1 Answers1

0

Do like this:

@transient val distinctBy = Window.partitionBy("id").orderBy(desc("updated_at"))

It's a long story. See Task not serializable exception while running apache spark job and https://medium.com/swlh/spark-serialization-errors-e0eebcf0f6e6 and https://nathankleyn.com/2017/12/29/using-transient-and-lazy-vals-to-avoid-spark-serialisation-issues/

thebluephantom
  • 16,458
  • 8
  • 40
  • 83