How to select list of specific columns (which contain special characters) from pyspark dataframe?

Question

I am trying to select specific columns from spark dataframe.

Specific columne list is:

required_cols = ['123ABC.PM','456DEF.PM']

Spark_df is in given format:

'123ABC.PM', '54SWC.PM', '456DEF.PM', '154AS.LB'
23.5         34.5         400.7        100.3
25.4         37.6         401          100
and so on

I have already tried:

spark_df_new = spark_df.select(required_cols)

But I am getting error:

"cannot resolve '`123ABC.PM`' given input columns: [123ABC.PM,54SWC.PM, 456DEF.PM,154AS.LB]
``

Possible duplicate of [Column name with dot spark](https://stackoverflow.com/questions/44367019/column-name-with-dot-spark) — pault, Jul 17 '19 at 15:34

score 1 · Accepted Answer · answered Jul 17 '19 at 07:09

1

Use back tick ` char

required_cols = [`123ABC.PM`,`456DEF.PM`]

answered Jul 17 '19 at 07:09

Ranga Vure

1,922
3
16
23

Back Quote works as well in Spark SQL `SELECT` statement if your column as a special meaning such as function name – Kiwy Nov 13 '20 at 15:25

score 0 · Answer 2 · answered Jul 17 '19 at 10:55

Probably as workaround you can try below approach.

Replace the old column names with special characters to new columns and then do a select.

val columns = df.columns

val regex = """[+._,' ]+"""
val replacingColumns = columns.map(regex.r.replaceAllIn(_, "_"))

val resultDF = replacingColumns.zip(columns).foldLeft(df){(tempdf, name) => tempdf.withColumnRenamed(name._2, name._1)}

resultDF.show(false)

or

df
  .columns
  .foldLeft(df){(newdf, colname) =>
    newdf.withColumnRenamed(colname, colname.replace(" ", "_").replace(".", "_"))
  }

Source: SO

score 0 · Answer 3 · answered Oct 07 '19 at 22:42

0

You need to use *.
* passes the elements in your list to the selection one by one.

spark_df_new = spark_df.select(*required_cols)

answered Oct 07 '19 at 22:42

Anuj Dutt

1

How to select list of specific columns (which contain special characters) from pyspark dataframe?

3 Answers3