I'm trying to generate a DataFrame in Spark (but perhaps just Scala is enough) in which I have every combination of the last 24 months where the second year-month is always > the first year-month.
For example, it is the 1 March 2019 as of writing this, I'm after something like:
List(
(2017, 3, 2017, 4),
(2017, 3, 2017, 5),
(2017, 3, 2017, 6),
// ..
(2017, 3, 2019, 3),
(2017, 4, 2017, 5),
// ..
(2019, 1, 2019, 3),
(2019, 2, 2019, 3),
)
This is easiest done with pure Scala without involving Spark. First, compute the list of all (year, month) tuples of the last 24 months. This can be done by using java.time and a Stream as follows:
import java.time.LocalDate
val numMonths = 24
val now = LocalDate.now()
val startTime = now.minusMonths(numMonths)
lazy val dateStream: Stream[LocalDate] = startTime #:: dateStream.map(_.plusMonths(1))
val dates = dateStream.take(numMonths + 1).toSeq.map(t => (t.getYear(), t.getMonth().getValue()))
Next, simply find all the 2-combinations of this tuple-sequence. This will automatically fulfill the condition that the second month should be after the first.
val datePerms = dates.combinations(2).map(c => (c(0)._1, c(0)._2, c(1)._1, c(1)._2))
You can easily convert this to a dataframe with the toDF method if necessary.
