Implicit resolution fail in reflection with ToolBox

Question

I am trying to generate Avro4s's RecordFormat in reflection based on class path. The following code throws an error.

case class MyCaseClass(a: Int)
println(toolBox.compile {
  toolBox.parse(
    s"""
       |import com.sksamuel.avro4s._
       |import mypackage.MyCaseClass
       |RecordFormat[MyCaseClass]
       |""".stripMargin
  )
}())

could not find implicit value for evidence parameter of type com.sksamuel.avro4s.Decoder[mypackage.MyCaseClass]

RecordFormat is like

object RecordFormat {

  def apply[T <: Product : Encoder : Decoder : SchemaFor]: RecordFormat[T] = apply(AvroSchema[T])

  def apply[T <: Product : Encoder : Decoder](schema: Schema): RecordFormat[T] = new RecordFormat[T] {
    private val fromRecord = FromRecord[T](schema)
    private val toRecord = ToRecord[T](schema)
    override def from(record: GenericRecord): T = fromRecord.from(record)
    override def to(t: T): Record = toRecord.to(t)
  }
}

Ref: https://github.com/sksamuel/avro4s/blob/release/2.0.x/avro4s-core/src/main/scala/com/sksamuel/avro4s/RecordFormat.scala

I can see, it can resolve Encoder[MyCaseClass] and SchemaFor[MyCaseClass] but fails for Decoder[MyCaseClass].

The same code can resolve RecordFormat[MyCaseClass] without reflection.

I can see that Decoder is implemented with macro similar to Encoder.

implicit def applyMacro[T <: Product]: Decoder[T] = macro applyMacroImpl[T]

Why reflection cannot resolve the implicit evidence?

I have +1000 case class in my class path defined in different packages and not maintained by me. My logic basically accepts the full path of the class for data processing. I am trying to use avro instead of json, and need to create `RecordFormat` from avro4s. Do you recommend more appropriate solution as RecordFormat highly depend on macros. — lalala, Nov 18 '20 at 23:53
@lalala Can't reproduce. `Decoder` seems to resolve as well as `SchemaFor` and `Encoder` https://scastie.scala-lang.org/HZR8Kf1WQkGtqOsc5efWuQ I made `mypackage` an object at Scastie but if it's a package, behavior locally is the same for me. Scala 2.13.3, avro4s 4.0.1. What is `apply` in the right hand side of the line inside the object `RecordFormat`? Now it seems to be the recursive call of `apply` from the left hand side, which can't be correct. — Dmytro Mitin, Nov 19 '20 at 02:44
@lalala As you can see at Scastie the implicits are resolved as `MagnoliaDerivedSchemaFors`, `MagnoliaDerivedEncoders`, `MagnoliaDerivedDecoders`. — Dmytro Mitin, Nov 19 '20 at 02:47
@DmytroMitin My Avro4s version is 2.x and Scala 2.11, I wonder if that is the reason. — lalala, Nov 19 '20 at 03:30
@DmytroMitin, regarding the recursive call, i have updated the question with the full library implementation and a reference link — lalala, Nov 19 '20 at 03:34

Dmytro Mitin · Accepted Answer · 2020-11-19T22:04:24.320

avro4s 4.x uses Magnolia but avro4s 2.x uses raw implicit macros + Shapeless.

Normally there shouldn't be significant problems with materializing a type class at runtime using reflective toolbox even if the type class is defined with macros.

The issue is now that the macro defining com.sksamuel.avro4s.Decoder has a bug. The line Decoder.scala#L404

c.Expr[Decoder[T]](
  q"""
  new _root_.com.sksamuel.avro4s.Decoder[$tpe] {
    private[this] val decoders = Array(..$decoders)

    override def decode(value: Any, schema: _root_.org.apache.avro.Schema): $tpe = {
      val fullName = $fullName
      value match {
        case record: _root_.org.apache.avro.generic.GenericRecord => $companion.apply(..$fields)
        case _ => sys.error("This decoder decodes GenericRecord => " + fullName + " but has been invoked with " + value)
      }
    }
  }
  """
)

refers to sys.error instead of hygienic _root_.scala.sys.error.

If you fix this line, Decoder[MyCaseClass] and RecordFormat[MyCaseClass] will work inside toolbox

println(toolBox.compile {
  toolBox.parse(
    s"""
       |import com.sksamuel.avro4s._
       |import mypackage.MyCaseClass
       |RecordFormat[MyCaseClass]
       |""".stripMargin
  )
}()) //com.sksamuel.avro4s.RecordFormat$$anon$1@25109d84

So a fast fix is to remove the line

libraryDependencies += "com.sksamuel.avro4s" %% "avro4s-core" % "2........."

in build.sbt, add

libraryDependencies += "com.chuusai" %% "shapeless" % "2.3.3"
libraryDependencies += "org.apache.avro" % "avro" % "1.8.2"

(otherwise you'll have NoClassDefFoundError) and put the following patched jars into lib

https://github.com/DmytroMitin/avro4s-2.0.5-2.11-patched

avro4s-core_2.11-2.0.5-SNAPSHOT.jar
avro4s-macros_2.11-2.0.5-SNAPSHOT.jar

You can always debug implicit-based or macro-based code generated with toolbox if you create it like

val toolBox = runtimeMirror.mkToolBox(
  frontEnd = new FrontEnd {
    override def display(info: Info): Unit = println(info)
    override def interactive(): Unit = ???
  },
  options = "-Xlog-implicits" // or "-Xlog-implicits -Ymacro-debug-lite"
)

If you do

println(reify{
  Decoder[MyCaseClass]
}.tree)

it prints

Decoder.apply[MyCaseClass](Decoder.applyMacro)

so implicit Decoder[MyCaseClass] is resolved as Decoder.applyMacro[MyCaseClass].

With the original unpatched jars

toolBox.compile {
  toolBox.parse(
    s"""
       |import com.sksamuel.avro4s._
       |import mypackage.MyCaseClass
       |Decoder.applyMacro[MyCaseClass]
       |""".stripMargin
  )
}()

threw

scala.tools.reflect.ToolBoxError: reflective compilation has failed:
object error is not a member of package sys

Great explanation! It is not directly related to my question but do you have any other solution than reflection. I do consume case classes from other packages. In runtime, I get a classpath as a configuration to build my logic. RecordFormat wants the type in compile time. Do you think it is possible to get RecordFormat from a class path instead of reflection based compiler. — lalala, Nov 19 '20 at 07:35
@lalala It's hard to say. I'm not sure I understand completely what prevents you from calling `RecordFormat[MyCaseClass]` directly (compile-time techniques would be type classes, implicits, macros, compile-time reflection). But if you really know a class name `MyCaseClass` only at runtime then resolving implicits with toolbox (or another way of running the compiler at runtime) seems the proper way. Otherwise you'll have to fill a `Map[String, RecordFormat[_]]` manually (where the string is a class name). — Dmytro Mitin, Nov 19 '20 at 12:07
@lalala Or from calling `RecordFormat[MyCaseClass]` if not directly then at least at compile time. — Dmytro Mitin, Nov 19 '20 at 12:24
@lalala Regarding having classpath as a configuration to build your logic. Some classes are known at compile time. The project should be organized as three subprojects: `common`, `macros` (depends on `common`), `core` (depends on `macros` and `common`). See [1](https://stackoverflow.com/questions/64472135/in-a-scala-macro-how-to-get-the-full-name-that-a-class-will-have-at-runtime) [2](https://stackoverflow.com/questions/63132189/def-macro-pass-parameter-from-a-value). Classes from `common` are known at compile time of `core` and can be used upon expanding macros from `macros`. — Dmytro Mitin, Nov 19 '20 at 12:31
@lalala So the question is whether you can do your logic at compile time. — Dmytro Mitin, Nov 19 '20 at 12:38

Implicit resolution fail in reflection with ToolBox

1 Answers1

Linked