Rewrite scala code to be more functional

Question

I am trying to teach myself Scala whilst at the same time trying to write code that is idiomatic of a functional language, i.e. write better, more elegant, functional code.

I have the following code that works OK:

import org.apache.spark.SparkConf
import org.apache.spark.sql.{DataFrame, SparkSession}
import java.time.LocalDate
object DataFrameExtensions_ {
  implicit class DataFrameExtensions(df: DataFrame){
    def featuresGroup1(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
    def featuresGroup2(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
  }
}
import DataFrameExtensions_._
val spark = SparkSession.builder().config(new SparkConf().setMaster("local[*]")).enableHiveSupport().getOrCreate()
import spark.implicits._
val df = Seq((8, "bat"),(64, "mouse"),(-27, "horse")).toDF("number", "word")
val groupBy = Seq("a","b")
val asAt = LocalDate.now()
val dataFrames = Seq(df.featuresGroup1(groupBy, asAt),df.featuresGroup2(groupBy, asAt))

The last line bothers me though. The two functions (featuresGroup1, featuresGroup2) both have the same signature:

scala> :type df.featuresGroup1(_,_)
(Seq[String], java.time.LocalDate) => org.apache.spark.sql.DataFrame

scala> :type df.featuresGroup2(_,_)
(Seq[String], java.time.LocalDate) => org.apache.spark.sql.DataFrame

and take the same vals as parameters so I assume I can write that line in a more functional way (perhaps using .map somehow) that means I can write the parameter list just once and pass it to both functions. I can't figure out the syntax though. I thought maybe I could construct a list of those functions but that doesn't work:

scala> Seq(featuresGroup1, featuresGroup2)
<console>:23: error: not found: value featuresGroup1
       Seq(featuresGroup1, featuresGroup2)
           ^
<console>:23: error: not found: value featuresGroup2
       Seq(featuresGroup1, featuresGroup2)
                           ^

Can anyone help?

Answer 1

I thought maybe I could construct a list of those functions but that doesn't work

You need to explicitly perform eta expansion to turn methods into functions (they are not the same in Scala), by using an underscore operator:

val funcs = Seq(featuresGroup1 _, featuresGroup2 _)

or by using placeholders:

val funcs = Seq(featuresGroup1(_, _), featuresGroup2(_, _))

And you are absolutely right about using map operator:

val dataFrames = funcs.map(f => f(groupBy, asAdt))

---

I strongly recommend against using implicits of types String or Seq, as if used in multiple places, these lead to subtle bugs that are not immediately obvious from the code and the code will be prone to breaking when it's moved somewhere.

If you want to use implicits, wrap them into a custom types:

case class DfGrouping(groupBy: Seq[String]) extends AnyVal

implicit val grouping: DfGrouping = DfGrouping(Seq("a", "b"))

Answer 2

I thought maybe I could construct a list of those functions but that doesn't work:

Why are you writing just featuresGroup1/2 here when you already had the correct syntax df.featuresGroup1(_,_) just above?

Seq(df.featuresGroup1(_,_), df.featuresGroup2(_,_)).map(_(groupBy, asAt))

df.featuresGroup1 _ should work as well.

df.featuresGroup1 by itself would work if you had an expected type, e.g.

val dataframes: Seq[(Seq[String], LocalDate) => DataFrame] = 
  Seq(df.featuresGroup1, df.featuresGroup2)

but in this specific case providing the expected type is more verbose than using lambdas.

Answer 3

Why no just create a function in DataFrameExtensions to do so?

def getDataframeGroups(groupBy: Seq[String], asAt: String) = Seq(featuresGroup1(groupBy,asAt), featuresGroup2(groupBy,asAt))

Answer 4

I think you could create a list of functions as below:

val funcs:List[DataFrame=>(Seq[String], java.time.LocalDate) => org.apache.spark.sql.DataFrame]  = List(_.featuresGroup1, _.featuresGroup1)
funcs.map(x => x(df)(groupBy, asAt))

It seems you have a list of functions which convert a DataFrame to another DataFrame. If that is the pattern, you could go a little bit further with Endo in Scalaz

Answer 5

I like this answer best, courtesy of Alexey Romanov.

import org.apache.spark.SparkConf
import org.apache.spark.sql.{DataFrame, SparkSession}
import java.time.LocalDate
object DataFrameExtensions_ {
  implicit class DataFrameExtensions(df: DataFrame){
    def featuresGroup1(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
    def featuresGroup2(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
  }
}
import DataFrameExtensions_._
val spark = SparkSession.builder().config(new SparkConf().setMaster("local[*]")).enableHiveSupport().getOrCreate()
import spark.implicits._
val df = Seq((8, "bat"),(64, "mouse"),(-27, "horse")).toDF("number", "word")
val groupBy = Seq("a","b")
val asAt = LocalDate.now()
Seq(df.featuresGroup1(_,_), df.featuresGroup2(_,_)).map(_(groupBy, asAt))