How to change date from yyyy-mm-dd to dd-mm-yyy using Spark function

Question

I'm working on Apache spark project on eclipse using Scala

I would like to change my date format from yyyy-mm-dd to dd-mm-yyyy

This is my code:

val conf = new SparkConf().setMaster("local").setAppName("trying")
val sc = new SparkContext(conf) 
val x = 
sc.textFile("/home/amel/1MB")
.filter(!_.contains("NULL")).filter(!_.contains("Null"))

val re = x.map(row => {
val cols = row.split(",")
val Cycle = cols(2)
val Duration = Cycle match {
case "Licence" => "3 years"
case "Master" => "2 years"
case "Ingéniorat" => "5 years"
case "Ingeniorat" => "5 years"
case "Doctorat" => "3 years"
case _ => "NULL" }
(cols(0)+","+cols(1) + "," + Cycle + "," +  cols(3) + "," 
+Duration)
})
re.collect.foreach(println)

This is an example of the result I got:

0000023497,2007-06-27,Master,SI,2 years

This is what I want my result to look like

0000023497,27-06-2007,Master,SI,2 years

Date parsing/formatting in Scala is usually done with `java.time`. [Here](https://stackoverflow.com/questions/18480633/java-util-date-format-conversion-yyyy-mm-dd-to-mm-dd-yyyy) is a close example. — Xavier Guihot, Jun 26 '19 at 19:49
Possible duplicate of [java.util.Date format conversion yyyy-mm-dd to mm-dd-yyyy](https://stackoverflow.com/questions/18480633/java-util-date-format-conversion-yyyy-mm-dd-to-mm-dd-yyyy) — Jonathan Myers, Jun 26 '19 at 20:30
Possible duplicate of [Convert date format in Scala](https://stackoverflow.com/questions/48411565/convert-date-format-in-scala) — Ole V.V., Jun 27 '19 at 08:35
Not a duplicate because of "using spark functions" clause in the question. Thanks. — C.S.Reddy Gadipally, Jun 27 '19 at 14:55
In this case: Possible duplicate of https://stackoverflow.com/questions/43348168/converting-pattern-of-date-in-spark-dataframe. Also the OP doesn't seem to be using a dataframe, which makes the other duplicates/solutions valid. — Xavier Guihot, Jun 27 '19 at 15:03
@XavierGuihot I agree. Duplicate of https://stackoverflow.com/questions/43348168/converting-pattern-of-date-in-spark-dataframe — C.S.Reddy Gadipally, Jun 27 '19 at 15:09

jwvh · Answer 1 · 2019-06-26T20:32:54.533

3

Can be done with regex.

val ymd = raw"(\d+)-(\d+)-(\d+)".r

ymd.replaceAllIn("2007-06-27", m => s"${m group 3}-${m group 2}-${m group 1}")
//res0: String = 27-06-2007

Can also be done via java.time library formatting.

import java.time.LocalDate
import java.time.format.DateTimeFormatter

LocalDate.parse("2019-01-04")
         .format(DateTimeFormatter.ofPattern("dd-MM-yyyy"))
//res1: String = 04-01-2019

edited Jun 26 '19 at 20:32

answered Jun 26 '19 at 20:25

jwvh

50,871
7
38
64

score 2 · Answer 2 · answered Jun 26 '19 at 22:37

Use org.apache.spark.sql.functions.date_format function

Example:

scala> df.show
+----------+
|      date|
+----------+
|2019-06-25|
|2019-06-26|
|2019-06-27|
+----------+

scala> df.withColumn("date2", org.apache.spark.sql.functions.date_format($"date", "dd-MM-yyyy")).show
+----------+----------+
|      date|     date2|
+----------+----------+
|2019-06-25|25-06-2019|
|2019-06-26|26-06-2019|
|2019-06-27|27-06-2019|
+----------+----------+

score 1 · Answer 3 · answered Jun 26 '19 at 20:43

1

Here is a Scala 2.13 alternative via pattern matching by unapplying a string interpolator:

"2007-06-27" match { case s"$year-$month-$day" => s"$day-$month-$year" }
// "27-06-2007"

answered Jun 26 '19 at 20:43

Xavier Guihot

54,987
21
291
190

How to change date from yyyy-mm-dd to dd-mm-yyy using Spark function

3 Answers3