Change column types iteratively Spark data frames

Question

I have a list of column names in Scala like

var cols = List("col1", "col2", "col3","col4")

Also I have a data frame with these columns but all in string. Now I would like to cast columns of dataframe by iterating through list or columns of dataframe because my list of columns is very large and I cant afford using so many .withColumn arguments

Thanks in Advance

Answer 1

If you know output types upfront it simply a matter of mapping over the columns with something similar to this

val df = sc.parallelize(Seq(
  ("foo", "1.0", "2", "true"),
  ("bar", "-1.0", "5", "false")
)).toDF("v", "x", "y", "z")

val types = Seq(
  ("v", "string"), ("x", "double"), ("y", "bigint"), ("z", "boolean")
)

df.select(types.map{case (c, t) => col(c).cast(t)}: _*)

If you don't know the types problem is much more trickier. While it is possible to create custom parser which can handle schema inference it probably make more sense to fix upstream pipeline instead. What is the point of using Avro when you ignore data types.

Answer 2

In case if you want to change multiple columns of a specific type to another without specifying individual column names. I have posted my answer here https://stackoverflow.com/a/60552157/3351492