Remove Blank Values from Spark DataFrame and merge into single row

Asked Jul 19 '18 at 09:11

Active Jul 19 '18 at 10:45

Viewed 41 times

This is my input DataFrame:

number|word|ID |Name|
+------+----+---+----+
|1     |    |   |    |
|      |abc |   |    |
|      |    |103|    |
|      |    |   |Dev |
+------+----+---+----

I want to remove the blank values and want to merge all the rows into single row like this:

number|word|ID |Name|
+------+----+---+----+
|1     | abc|103| Dev

First of all is it possible in spark? If yes then what is the best way to achieve. I am using Scala and Spark 2.1. Also, I can have n no.of columns.

edited Jul 19 '18 at 10:45

asked Jul 19 '18 at 09:11

Naman Agarwal

is that the only data? if you have more data in the input then please update the question with some more sample data in the input and desired expected output? – Ramesh Maharjan Jul 19 '18 at 09:20
As of now I have this as a sample data. My main motive is to remove the blank values from every row. – Naman Agarwal Jul 19 '18 at 09:24
1

I think a different type of the dataset would explain it better. What if there's only one blank field in the entire row or all blank fields. Your example isn't complete. try to provide a [mcve] – philantrovert Jul 19 '18 at 09:25
for every row there will be only on column populated and rest are blank. – Naman Agarwal Jul 19 '18 at 09:58
@user8371915 what if I have n no.of columns ?? – Naman Agarwal Jul 19 '18 at 10:34
Then [SparkSQL: apply aggregate functions to a list of column](https://stackoverflow.com/q/33882894/8371915) – Alper t. Turker Jul 19 '18 at 10:53
@user8371915 the solution works when values are null not with the blank values. Is there a way where I can convert the blank values to null ?? – Naman Agarwal Jul 19 '18 at 11:47
Try `first(when(trim(col(c)) =!= "", col(c)), true)` – Alper t. Turker Jul 19 '18 at 12:23

Remove Blank Values from Spark DataFrame and merge into single row

0 Answers0