Using scala macro to manipulate variable declaration

Question

i would like to use scala (v2.12.8) macros to manipulate all variable declarations of an given block. In this example to add the value 23.

For example:

val myblock = mymanipulator {
    var x = 1
    x = 4
    var y = 1
    x + y
  }
print( myblock )

becomes

{
  var x = (1).+(23);
  x = 4;
  var y = (1).+(23);
  x.+(y)
}

For this, I implemented mymanipulator like this:

import scala.language.experimental.macros
import scala.reflect.macros.blackbox.Context
import scala.language.implicitConversions

object mymanipulator {
  def apply[T](x: => T): T = macro impl

  def impl(c: Context)(x: c.Tree) = { import c.universe._

    val q"..$stats" = x
    val loggedStats = stats.flatMap { stat =>

      stat match {
        case ValDef(mods, sym, tpt, rhs) => {
          List( q"var $sym : $tpt = $rhs + 23" )
        }

        case _ => {
          List( stat )
        }
      }

    }

    val combined = q"..$loggedStats"

    c.info(c.enclosingPosition, "combined: " + showRaw(combined), true)

    combined
  }
}

And I get this information during compilation of the macro:

Information:(21, 31) combined: {
  var x = (1).+(23);
  x = 4;
  var y = (1).+(23);
  x.+(y)
}
  val myblock = mymanipulator {

But when I execute the mymanipulator with the given block above, then I get this error message:

Error:scalac: Error while emitting Test.scala
variable y

This error occurs also when I change the implementation to do nothing:

 stat match {
        case ValDef(mods, sym, tpt, rhs) => {
          List( q"var $sym : $tpt = $rhs" )
        }

        case _ => {
          List( stat )
        }
      }

Only when I return stat the error vanishes

 stat match {
    case ValDef(mods, sym, tpt, rhs) => {
       List( stat )
     }

     case _ => {
        List( stat ) 
      }
  }

Can somebody tell me that I am doing wrong? Thanks

Dmytro Mitin · Accepted Answer · 2019-03-08T20:32:30.100

You should untypecheck the tree before transformations

  object mymanipulator {
    def apply[T](x: => T): T = macro impl

    def impl(c: blackbox.Context)(x: c.Tree): c.Tree = {
      import c.universe._

      val q"..$stats" = c.untypecheck(x) // here
      val loggedStats = stats.flatMap { stat =>

        stat match {
          case ValDef(mods, sym, tpt, rhs) /*q"$mods var $sym : $tpt = $rhs"*/ => {
            List( q"$mods var $sym : $tpt = $rhs + 23" )
          }

          case _ => {
            List( stat )
          }
        }

      }

      val combined = q"..$loggedStats"

      c.info(c.enclosingPosition, "combined: " + showRaw(combined), true)

      combined
    }
  }

Macro untypecheck required

What is wrong with this def macro?

Serhiy Shamshetdinov · Answer 2 · 2021-03-16T11:06:19.470

Untypechecking the source tree drops the types of subtrees. This makes impossible to do tree manipulating based on types of expressions. So how to work with typechecked tree and replace the terms definitions in the macro source code?

If we replace the definition of term (or even simply reassemble the same term definition, actually having new tree) then compiler fails on Ident of that term with error like:

Could not find proxy for val <val-name>

REPL error is like

Error while emitting <console>
variable j

Simple untypechecking or even further typechechking of the resulting tree does not helps. There are several reasons I've found in different answers:

Old (reused from source tree) Ident still refers to it's old Symbol definition that is already is absent in resulting tree (we replaced it)
The reused unchanged Symbol definition (its tree) has changed its owner (the code was wrapped or rewrapped)

The solution that helps me is to recreate all Idents which refers to local (to source code) definitions. Idents that refer to outer Symbols definitions should stay unchanged (like 'scala', internally referred outer types, etc.) otherwise compiling fails.

The following sample (runnable in IDEA Worksheet in REPL mode, tried in 2.12) shows usage of Transformer to recreate Idents that refer to local definition only. New replaced Ident will not now refer to old definition.

It uses syntax tree that covers the only required Scala syntax to reach the goal. Everything unknow by this syntax becomes OtherTree that holds subtree of original source code.

import scala.reflect.macros.blackbox
import scala.language.experimental.macros

trait SyntaxTree {
  val c: blackbox.Context
  import c.universe._

  sealed trait Expression

  case class SimpleVal(termName: TermName, expression: Expression) extends Expression
  case class StatementsBlock(tpe: Type, statements: Seq[Expression], expression: Expression) extends Expression
  case class OtherTree(tpe: Type, tree: Tree) extends Expression

  object Expression {
    implicit val expressionUnliftable: Unliftable[Expression] = Unliftable[Expression] (({
      case q"val ${termName: TermName} = ${expression: Expression}" =>
        SimpleVal(termName, expression)
      case tree@Block(_, _) => // matching on block quosiquotes directly produces StackOverflow in this Syntax: on the single OtherTree node
        val q"{ ..${statements: Seq[Expression]}; ${expression: Expression} }" = tree
        StatementsBlock(tree.tpe, statements, expression)
      case otherTree =>
        OtherTree(otherTree.tpe, otherTree)
    }: PartialFunction[Tree, Expression]).andThen(e => {println("Unlifted", e); e}))

    implicit val expressionLiftable: Liftable[Expression] = Liftable[Expression] {
      case SimpleVal(termName, expression) =>
        q"val $termName = $expression + 23"
      case StatementsBlock(_, statements, expression) =>
        q"{ ..${statements: Seq[Expression]}; ${expression: Expression} }"
      case OtherTree(_, otherTree) =>
        c.untypecheck(otherTree) // untypecheck here or before final emitting of the resulting Tree: fun, but in my complex syntax tree this dilemma has 0,01% tests impact (in both cases different complex tests fails in ToolBox)
    }
  }
}

class ValMacro(val c: blackbox.Context) extends SyntaxTree {
  import c.universe._

  def valMacroImpl(doTransform: c.Expr[Boolean], doInitialUntypecheck: c.Expr[Boolean])(inputCode: c.Expr[Any]): c.Tree = {
    val shouldDoTransform = doTransform.tree.toString == "true"
    val shouldUntypecheckInput = doInitialUntypecheck.tree.toString == "true"

    val inputTree = if (shouldUntypecheckInput)
      c.untypecheck(inputCode.tree) // initial untypecheck helps but we loose parsed expression types for analyses
    else
      inputCode.tree

    val outputTree: Tree = inputTree match {
      case q"${inputExpression: Expression}" =>
        val liftedTree = q"$inputExpression"
        if (shouldDoTransform) {
          val transformer = new LocalIdentsTransformer(inputTree)
          transformer.transform(liftedTree)
        } else
          liftedTree
      case _ =>
        q"{ ${"unexpected input tree"} }"
    }
    println(s"Output tree: $outputTree")
    /*c.typecheck(c.untypecheck(*/outputTree/*))*/ // nothing commented helps without transforming (recreating) Idents
  }

  class LocalIdentsTransformer(initialTree: Tree) extends Transformer {
    // transform is mandatory in any case to relink (here: reset) Ident's owners when working with typechecked trees
    private val localDefSymbols: Set[Symbol] = initialTree.collect {
      case t if t != null && t.isDef && t.symbol.isTerm =>
        t.symbol
    }.toSet

    println("localDefSymbols", localDefSymbols)

    override def transform(tree: Tree): Tree = tree match {
      case tree@Ident(termName: TermName) if localDefSymbols.contains(tree.symbol) =>
        println("replacing local Ident", termName, tree.symbol)
        Ident(termName)
      case _ =>
        super.transform(tree)
    }
  }
}

def valMacro(doTransform: Boolean, doInitialUntypecheck: Boolean)(inputCode: Any): Any = macro ValMacro.valMacroImpl

val outerVal = 5
// 1) works with pre untypechecking, but we loose types info
valMacro(false, true) {
  val i = 1
  i + outerVal
}
// 2) works with Transformer
valMacro(true, false) {
  val i = 1
  i + outerVal
}
// 3) does not work
valMacro(false, false) {
  val i = 1
  i + outerVal
}
// 4) cases when we reuse old tree without changes: fails
valMacro(false, false) {
  var j = 1
  j
}
// 5) cases when we reuse old tree without changes: works
valMacro(true, false) {
  var j = 1
  j
}

Output:

// 1) works with pre untypechecking, but we loose types info
(Unlifted,OtherTree(null,1))
(Unlifted,SimpleVal(i,OtherTree(null,1)))
(Unlifted,OtherTree(null,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)))
(Unlifted,StatementsBlock(null,List(SimpleVal(i,OtherTree(null,1))),OtherTree(null,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal))))
Output tree: {
  val i = 1.$plus(23);
  i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)
}
res0: Any = 29

// 2) works with Transformer
Unlifted,OtherTree(Int(1),1))
(Unlifted,SimpleVal(i,OtherTree(Int(1),1)))
(Unlifted,OtherTree(Int,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)))
(Unlifted,StatementsBlock(Int,List(SimpleVal(i,OtherTree(Int(1),1))),OtherTree(Int,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal))))
(localDefSymbols,Set(value i))
(replacing local Ident,i,value i)
Output tree: {
  val i = 1.$plus(23);
  i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)
}
res1: Any = 29

// 3) does not work
(Unlifted,OtherTree(Int(1),1))
(Unlifted,SimpleVal(i,OtherTree(Int(1),1)))
(Unlifted,OtherTree(Int,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)))
(Unlifted,StatementsBlock(Int,List(SimpleVal(i,OtherTree(Int(1),1))),OtherTree(Int,i.+($line8.$read.INSTANCE.$iw.$iw.outerVal))))
Output tree: {
  val i = 1.$plus(23);
  i.+($line8.$read.INSTANCE.$iw.$iw.outerVal)
}

Error while emitting <console>
value i

// 4) case when we reuse old tree without changes: fails
(Unlifted,OtherTree(<notype>,var j: Int = 1))
(Unlifted,OtherTree(Int,j))
(Unlifted,StatementsBlock(Int,List(OtherTree(<notype>,var j: Int = 1)),OtherTree(Int,j)))
Output tree: {
  var j = 1;
  j
}

Error while emitting <console>
variable j

// 5) case when we reuse old tree without changes: works with Transformer
(Unlifted,OtherTree(<notype>,var j: Int = 1))
(Unlifted,OtherTree(Int,j))
(Unlifted,StatementsBlock(Int,List(OtherTree(<notype>,var j: Int = 1)),OtherTree(Int,j)))
(localDefSymbols,Set(variable j))
(replacing local Ident,j,variable j)
Output tree: {
  var j = 1;
  j
}

If to skip untypchecking of OtherTree (while lifting it) or do not do untypchecking of theresulting tree then we'll get the error in 2) sample macro call:

java.lang.AssertionError: assertion failed: 
  transformCaseApply: name = i tree = i / class scala.reflect.internal.Trees$Ident
     while compiling: <console>
        during phase: refchecks
     library version: version 2.12.12
    compiler version: version 2.12.12

Actually this sample show 2 different approaches to work with typechecked input tree:

How to use simple AST that covers only the part of Scala syntax tree with required types collection (it is actually may not be used). If such AST is not used and resulting tree is built "on the fly" using parts of the source typechecked tree (resulting partially typechecked tree) then resulting tree should be untypechecked (after transformation) before emitting
How to fix Idents in partially typechecked resulting tree with LocalIdentsTransformer

Using scala macro to manipulate variable declaration

2 Answers2

Linked