How to concat a String to the column of a Dataframe in spark?

Question

I have a dataframe: precisionCols, with one column.

val spColsDF = spark.read.format("jdbc").option("url",hiveMetaConURL)
                                               .option("dbtable", "(select source_columns, precision_columns from bdmerge.scope_tables where tablename='schema.casting') as colsPrecision")
                                               .option("user", metaUserName)
                                               .option("password", metaPassword)
                                               .load()
val precisionCols       = spColsDF.select("precision_columns")

There are only three rows in the dataframe:

precisionCols.collect.foreach(println) => [ptd_balance,qtd_balance,ytd_balance]

I need to concat the column values with a String "_text" and change them to ptd_balance_text,qtd_balance_text,ytd_balance_text

I tried the below ways but I am doing it wrong somehow:

precisionCols.withColumn("precision_columns_text", precisionCols("precision_columns") + lit("_text"))
precisionCols.withColumn("precision_columns_text", precisionCols("precision_columns") + "_text")

Input Data:

-----------------
precision_columns
-----------------
ptd_balance
qtd_balance
ytd_balance

The output comes as:

[ptd_balance,qtd_balance,ytd_balance_text]

The output I am trying to get is:

-----------------
precision_columns
-----------------
ptd_balance_text
qtd_balance_text
ytd_balance_text

Could anyone let me know how to concat a String to existing columns to create a new DataFrame ?

I don think its a duplicate. The question asked is concatenating a string to the column and not concatenating 2 columns — Gangadhar Kadam, Aug 05 '18 at 14:08
so if a row has a, b, x, in col1, col2, col3 --> a,b,c_text is the value under which colname on the new dataframe? or existing with withCol — thebluephantom, Aug 05 '18 at 18:30
There is only one column: "precision_columns", with three rows in it. I am trying to concatenate "_text" to all three rows. — Metadata, Aug 05 '18 at 18:35
wondering if you mean pivot? sorry a little hard to follow, you show only one "text" concatenated but above that all three. Do you want to go from 1 col to 3 cols? — thebluephantom, Aug 05 '18 at 18:46
Yes. I have added the expected output. It should give a better explanation. — Metadata, Aug 05 '18 at 18:51
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/177453/discussion-between-thebluephantom-and-sidhartha). — thebluephantom, Aug 05 '18 at 20:23

How to concat a String to the column of a Dataframe in spark?

0 Answers0