How does `as 'value` work in Spark in Scala?

Question

When I use Spark in Scala, I found three ways to do same thing (?)

.select(to_avro(struct("*"), toAvroConfig).alias("value"))
.select(to_avro(struct("*"), toAvroConfig).as("value"))
.select(to_avro(struct("*"), toAvroConfig) as 'value)

I found documents about first two:

alias doc:

  def alias(alias: String): Column = name(alias)

  /**
   * Gives the column an alias.
   * {{{
   *   // Renames colA to colB in select output.
   *   df.select($"colA".as("colB"))
   * }}}
   *
   * If the current column has metadata associated with it, this metadata will be propagated
   * to the new column. If this not desired, use the API `as(alias: String, metadata: Metadata)`
   * with explicit metadata.
   *
   * @group expr_ops
   * @since 1.3.0
   */

as doc:

  def as(alias: String): Column = name(alias)

  /**
   * (Scala-specific) Assigns the given aliases to the results of a table generating function.
   * {{{
   *   // Renames colA to colB in select output.
   *   df.select(explode($"myMap").as("key" :: "value" :: Nil))
   * }}}
   *
   * @group expr_ops
   * @since 1.4.0
   */

For last one as 'value, I found it here. Initially I thought it is a typo, but it is actually valid. How does it work? Is there a document about it?

Are all three same? Thanks!

`'value` is deprecated syntax for a `Symbol` and either there is an overload or some implicit conversion from `Symbol` to `String`, or maybe the compiler treats the subtypes since symbols are just constant strings. - anyways, that thing is not used anymore in modern **Scala** _(although **Spark** is far from that anyways)_ — Luis Miguel Mejía Suárez, May 02 '23 at 23:54

score 3 · Accepted Answer · answered May 03 '23 at 04:38

The scaladoc that correspond to the method, is the one that is above. So, the doc for def alias(alias: String): Column = name(alias) is this and the doc for def as(alias: String): Column = name(alias) is this. As you can see in the scaladoc of alias says

Gives the column an alias. Same as as.

So, both methods do the same.

About how 'value work? You can have a good explanation in the post What are some example use cases for symbol literals in Scala?. These feature is deprecated as you can see here https://docs.scala-lang.org/scala3/guides/migration/incompat-dropped-features.html#symbol-literals

How does `as 'value` work in Spark in Scala?

1 Answers1