How to use a `universe.Tree` created and type-checked in one Scala macro execution, in another macro execution?

Question

In the JSON library jsfacile I was able to automatically derive type-classes for types with recursive type references thanx to a buffer where outer executions of a macro store instances of universe.Tree which are used later by inner executions of the same or other macro. This works fine as long as the universe.Tree instance was not type-checked (with the Typers.typecheck method).

The problem is that this forces to type-check the same Tree instance more than one time: one time in the macro execution that created it (after having stored it in the buffer); and more times in each inner macro execution that needs it.

The intention of this question is to find out a way to share the Tree instance between macro executions after it was type-checked; in order to improve compilation speed.

I tried to wrap the type-checked Tree into a universe.Expr[Unit] and migrate it to the mirror of the macro execution that uses it by means of the Expr.in[U <: Universe](otherMirror: Mirror[U]): U # Expr[T] method. But it fails with an internal error:

Internal error: unable to find the outer accessor symbol of class <Name of the class where the macro is expanded>

Any idea?

Regarding sharing a typed tree among macro expansions. Is it always a good idea? The same tree can be typechecked in different contexts differently (implicits, names etc.). For example `SomeName` can be typechecked as `onepackage.SomeName` in one context and `anotherpackage.SomeName` in another. — Dmytro Mitin, Nov 22 '20 at 04:52
@DmytroMitin Perhaps the question isn't clear. The the macro executions occur at the same code position. They are all part of the same expansion, but different deep of recursion. The context is the same. — Readren, Nov 22 '20 at 06:56
And even if the context were different I think the sharing would work anyway if the buffer where the `Tree` instances are stored was indexed by the type for which the type-class is derived to (the buffer keys equality is defined using the `=:=` operator). If I am not wrong, the relation between the type for which the type-class is derived to and the stored `Tree` instances is one to one independently of the context. If said type depended on some name or implicit, it would be a different type in terms of the `=:=` operator. — Readren, Nov 22 '20 at 07:30
@DmytroMitin The macros of my library I am talking about are the ones defined in `ProductParserHelper` and `CoproductParserHelper`. Note that they work fine as they are in the `origin/master` repository. Because in that version the `Tree` instances are stored before the type-checking. If you are brave enough to try to understand my cryptic code, I will gladly create a branch with the change causing the error. — Readren, Nov 22 '20 at 07:45
Done. The branch "typecheckedSharing" has the changes causing the internal error. Note that the mutual recursion of the `ProductParserHelper` and `CoproductParserHelper` macros is not explicit. It is triggered by the `typecheck` method when the `Tree` instance contains implicit parameters that trigger the execution of one said of macros. The error occurs when the `ParserMacroTest` or `AppenderMacroTest` are compiled. — Readren, Nov 22 '20 at 08:23
Why do you typecheck trees manually? Do you know that typechecking is not idempotent? https://github.com/scala/bug/issues/5464 — Dmytro Mitin, Nov 22 '20 at 13:24
*"the macro executions occur at the same code position"* The same position doesn't mean the same context. Typechecking `implicitly[TC[Int]]` in the context `implicit def tc[A]: TC[A] = null; implicitly[TC[Int]]` and in the context `implicit val i: TC[Int] = null; { implicit def tc[A]: TC[A] = null; implicitly[TC[Int]] }` are different. — Dmytro Mitin, Nov 22 '20 at 13:58
*"And even if the context were different I think the sharing would work anyway if the buffer where the `Tree` instances are stored was indexed by the type..."* Typechecking changes not only `Type` field of a `Tree` but also its `Symbol` field. I suspect that with your manipulations on trees, manual typechecking, caching trees among different contexts during implicit resolution etc. you could break somewhere symbol chains. "Internal error: unable to find the outer accessor symbol" can signal about that. Please read https://github.com/scalamacros/macrology201 for better understanding. — Dmytro Mitin, Nov 22 '20 at 14:05
By the way, you have macros both in `macros` and `core`. Is it intended? — Dmytro Mitin, Nov 22 '20 at 15:48
@DmytroMitin "Why do you type-check trees manually?" I type-check `Tree` instances manually to trigger the execution of the inner macros. That way a outer macro execution knows (after the type-check) all the entries in the buffer that were added by the inner macro executions. — Readren, Nov 22 '20 at 16:43
@DmytroMitin "you have macros both in macros and core. Is it intended?" There are no macro implementations in the `core` module. Only macro calls. Is that a problem? — Readren, Nov 22 '20 at 16:44
Standard approach is to duplicate a tree before typechecking (`c.typecheck(tree.duplicate)`). You can try to return untyped trees and rely on compiler typechecking after a macro is expanded (then symbol chain will be calculated: `symbol`, `symbol.owner`, `symbol.owner.owner`, ..., `ctx.mirror.RootClass`). Mixing typed and untyped trees can sometimes be dangerous. If the compiler sees a typed node it assumes that its children are typed too and don't retypecheck them (so absent or wrong symbol owners are not restored). See details in macrology201. — Dmytro Mitin, Nov 22 '20 at 18:08
https://stackoverflow.com/questions/20936509/scala-macros-what-is-the-difference-between-typed-aka-typechecked-an-untyped — Dmytro Mitin, Nov 22 '20 at 18:44
*"There are no macro implementations in the core module. Only macro calls. Is that a problem?"* No. Just `-language:experimental.macros` has to be visible both in `macros` and `core`. — Dmytro Mitin, Nov 23 '20 at 01:28

Dmytro Mitin · Accepted Answer · 2020-11-25T19:02:59.490

Generally, typechecking a tree manually and sharing the typed tree among different contexts is a bad idea. See the following example:

import scala.language.experimental.macros
import scala.reflect.macros.whitebox
import scala.collection.mutable

object Macros {
  val mtcCache = mutable.Map[whitebox.Context#Type, whitebox.Context#Tree]()

  trait MyTypeclass[A]

  object MyTypeclass {
    implicit def materialize[A]: MyTypeclass[A] = macro materializeImpl[A]

    def materializeImpl[A: c.WeakTypeTag](c: whitebox.Context): c.Tree = {
      import c.universe._

      val typA = weakTypeOf[A]

      if (!mtcCache.contains(typA)) 
        mtcCache += (typA -> c.typecheck(q"new MyTypeclass[$typA] {}"))

      mtcCache(typA).asInstanceOf[Tree]
    }
  }
}

import Macros.MyTypeclass

object App { // Internal error: unable to find the outer accessor symbol of object App

  class A { // Internal error: unable to find the outer accessor symbol of class A

    class B { // Internal error: unable to find the outer accessor symbol of class A

      class C {
        implicitly[MyTypeclass[Int]] // new MyTypeclass[Int] {} is created and typechecked here
      }

      implicitly[MyTypeclass[Int]] // cached typed instance is inserted here, this is the reason of above error
    }

    implicitly[MyTypeclass[Int]] // cached typed instance is inserted here, this is the reason of above error
  }

  implicitly[MyTypeclass[Int]] // cached typed instance is inserted here, this is the reason of above error
}

Scala 2.13.3.

With implicitly we put in some places trees with incorrect symbol owner chain.

If you make A, B, C objects then errors disappear (so whether this prevents compilation depends on a luck).

Also if you remove c.typecheck then errors disappear.

Also if we return c.untypecheck(mtcCache(typA).asInstanceOf[Tree]) instead of mtcCache(typA).asInstanceOf[Tree] then errors disappear. But sometimes c.typecheck + c.untypecheck can damage a tree.

So you can try to put both untyped and typed versions of a tree to the cache if you need both but return the untyped one

type CTree = whitebox.Context#Tree
val mtcCache = mutable.Map[whitebox.Context#Type, (CTree, CTree)]()

trait MyTypeclass[A]

object MyTypeclass {
  implicit def materialize[A]: MyTypeclass[A] = macro materializeImpl[A]

  def materializeImpl[A: c.WeakTypeTag](c: whitebox.Context): c.Tree = {
    import c.universe._

    val typA = weakTypeOf[A]
    val tree = q"new MyTypeclass[$typA] {}"
    if (!mtcCache.contains(typA)) 
      mtcCache += (typA -> (tree, c.typecheck(tree)))

    mtcCache(typA)._1.asInstanceOf[Tree]
  }
}

or if you need typechecking only to trigger the recursion then you can typecheck a tree, put the untyped tree to the cache and return the untyped one

val mtcCache = mutable.Map[whitebox.Context#Type, whitebox.Context#Tree]()

trait MyTypeclass[A]

object MyTypeclass {
  implicit def materialize[A]: MyTypeclass[A] = macro materializeImpl[A]

  def materializeImpl[A: c.WeakTypeTag](c: whitebox.Context): c.Tree = {
    import c.universe._

    val typA = weakTypeOf[A]
    val tree = q"new MyTypeclass[$typA] {}"
    c.typecheck(tree)
    if (!mtcCache.contains(typA)) mtcCache += (typA -> tree)

    mtcCache(typA).asInstanceOf[Tree]
  }
}

Pull request: https://github.com/readren/json-facile/pull/1

There you are not migrating the tree to the current context with `Expr.in[U <: Universe](otherMirror: Mirror[U]): U # Expr[T]` as I explained in the question. But anyway, even if you did it, the compiler would complains with the same error. I tested that based on your code. So the migration changes nothing. — Readren, Nov 23 '20 at 02:13
@Readren If you look how `in` is [implemented](https://github.com/scala/scala/blob/2.13.x/src/reflect/scala/reflect/api/Exprs.scala#L147-L151) you'll see that it's actually several `asInstanceOf`. I prefer to work with `Tree`s rather than `Expr`s. — Dmytro Mitin, Nov 23 '20 at 02:17
@Readren So replacing `asInstanceOf` with `Expr#in` can't make a difference. You can consider `Expr[A](tree)` as a "tuple" of `A` and `tree`. `tree` can be typed (`c.Expr[Int](c.typecheck(q"1+1"))`) or untyped (`c.Expr[Int](q"1+1")`). Your mistake was abusing typechecking. — Dmytro Mitin, Nov 23 '20 at 02:37
In the `master` I use `Tree` instances directly, which are shared unchecked. The incorporation of `Expr` was in the `typecheckedSharing` branch I made for you, only because it has the `in` method. I incorporated it because the `in` documentation promises it migrates a `Expr` from one context to another, but that's not true according to your comments and the results. I wont have asked this SO question in the first place if the `in` method didn't exist. Next time I will inspect the implementation. — Readren, Nov 23 '20 at 03:52
By the way, your ability to reproduce errors is surprising. Your help was enormous. — Readren, Nov 23 '20 at 04:18

How to use a `universe.Tree` created and type-checked in one Scala macro execution, in another macro execution?

1 Answers1

Linked