1

I've defined a case class to be used as a schema for a Dataset in Spark.

I want to be able to refer to individual columns from that schema by referencing them programmatically (vs. hardcoding their string value somewhere)

For example, for the following case class

final case class MySchema(id: Int, name: String, timestamp: Long)

I would like to auto-generate the following object

object MySchema {

val id = "id"

val name = "name"

val timestamp = "timestamp"

}

The Macro approach outlined here appears to be what I want, but it won't compile under Scala 2.12. It gives the following errors which are completely baffling to me and show up in a total of 2 Google results with 0 fixes.

[error] pattern var qq$macro$2 in method unapply is never used: use a wildcard `_` or suppress this warning with `qq$macro$2@_`
[error]       case (c@q"$_ class $tpname[..$_] $_(...$params) extends { ..$_ } with ..$_ { $_ => ..$_ }") :: Nil =>
[error]               ^
[error] pattern var qq$macro$19 in method unapply is never used: use a wildcard `_` or suppress this warning with `qq$macro$19@_`
[error]       case (c@q"$_ class $_[..$_] $_(...$params) extends { ..$_ } with ..$_ { $_ => ..$_ }") ::
[error]               ^
[error] pattern var qq$macro$27 in method unapply is never used: use a wildcard `_` or suppress this warning with `qq$macro$27@_`
[error]         q"$mods object $tname extends { ..$earlydefns } with ..$parents { $self => ..$body }" :: Nil =>
[error]         ^

Suppressing the warning as outlined won't work because the macro numbers change every time I compile.

It's also worth noting that the similar SO answer here runs into the same compiler errors as shown above

IntelliJ also complains about several parts of the macro that the compiler doesn't complain about, but that's not really an issue if I can get it to compile

Is there a way to fix that Macro approach to work in Scala 2.12 or is there a better Scala 2.12 way to do this? (I can't use Scala 2.13 or higher due to compute environment constraints)

oey192
  • 13
  • 4

1 Answers1

1

Just checked that the macro is still working both in Scala 2.13.10 and 2.12.17.

Most probably, you didn't set up your project for macro annotations

build.sbt

//ThisBuild / scalaVersion := "2.13.10"
ThisBuild / scalaVersion := "2.12.17"

lazy val macroAnnotationSettings = Seq(
  scalacOptions ++= (CrossVersion.partialVersion(scalaVersion.value) match {
    case Some((2, v)) if v >= 13 => Seq("-Ymacro-annotations") // for Scala 2.13
    case _ => Nil
  }),
  libraryDependencies ++= (CrossVersion.partialVersion(scalaVersion.value) match {
    case Some((2, v)) if v <= 12 => // for Scala 2.12
      Seq(compilerPlugin("org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.full))
    case _ => Nil
  })
)

lazy val core = project
  .settings(
    macroAnnotationSettings,
    scalacOptions ++= Seq(
      "-Ymacro-debug-lite", // optional, convenient to see how macros are expanded
    ),
  )
  .dependsOn(macros) // you must split your project into subprojects because macros must be compiled before core

lazy val macros = project
  .settings(
    macroAnnotationSettings,
    libraryDependencies ++= Seq(
      scalaOrganization.value % "scala-reflect" % scalaVersion.value, // necessary for macros
    ),
  )

project structure:

core
  src
    main
      scala
        Main.scala
macros
  src
    main
      scala
        Macros.scala

Then just do sbt clean compile.

The whole project: https://gist.github.com/DmytroMitin/2d9dbd6441ebf167aa127b80fb516afd

sbt documentation: https://www.scala-sbt.org/1.x/docs/Macro-Projects.html

Scala documentation: https://docs.scala-lang.org/overviews/macros/annotations.html

Examples of build.sbt:

https://github.com/typelevel/simulacrum/blob/master/build.sbt

https://github.com/DmytroMitin/AUXify/blob/master/build.sbt

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
  • I haven't had time to verify if your answer fully fixes my issue, but I wasn't aware I had to do anything special to enable macros in Scala, so you're correct that I will need to do so. At this point I've switched to use DataFrame instead of Dataset so I don't need the functionality described in the question anymore, but if I need it in the future I'll give this a shot. Thanks! – oey192 Jan 07 '23 at 00:43