0

Issue

I have a scenario where I need to convert a scala Map to a case class object and with help of the following references I was able to achieve it locally (scala version 2.12.13):

  1. Scala: convert map to case class
  2. Convert a Map into Scala object

But when I tried running the same block of code in Databricks notebook it throws an error:

IllegalArgumentException: Cannot construct instance of '$line23851bc084ae4df7a16bf9c475868d9265.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$Test' (although at least one Creator exists): can only instantiate non-static inner class by using default, no-argument constructor at [Source: UNKNOWN; line: -1, column: -1]

Cluster configuration: Databricks runtime 8.2(Includes Spark 3.1.1, Scala 2.12). Please refer to the screenshot for the complete code.

Workaround (not advisable):

def workaround(map: Map[String, Any]): Test = {
  Test(
    map("k1").asInstanceOf[Int],
    map("k2").asInstanceOf[String],
    map("k3").asInstanceOf[String],
  )
}

val result = workaround(myMap)

Any thoughts on how to resolve this issue?

Sameer Kumar
  • 11
  • 5

1 Answers1

0

This smells to me like one of two possibilities.

First, we should double check that there is no version mis-matching between something in your local runtime environment and the databricks runtime environment. You can check this page for a list of all library versions included in DBR 8.2. In particular, I would check your local environment to make sure you are running the same version of jackson (2.10.0).

Second, this could be an interaction between how Databricks implements their notebook and a limitation of jackson. Each command of a databricks notebooks is wrapped in a package object that is given a random name. For example, I can tell from the exception that the package object which holds your Test class definition was called $line23851bc084ae4df7a16bf9c475868d9265.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw. (I say that it was named that because each time you re-run the command it will generate a new package object). This implies that all classes (and other kinds of type definitions) are actually path dependent types when put inside a notebook. More specifically for this scenario, your Test class is an inner-class to the package object. based on the error message and some brief reading of docs, I suspect that Jackson cannot serialize path dependent types.

Erp12
  • 600
  • 3
  • 16