How to split RDD rows by commas when there is no value between them?

Question

I'm trying to split the below RDD row into five columns

val test = [hello,one,,,]

val rddTest = test.rdd
val Content = rddTest.map(_.toString().replace("[", "").replace("]", ""))
      .map(_.split(","))
      .map(e ⇒ Row(e(0), e(1), e(2), e(3), e(4), e(5)))

when I execute I get "java.lang.ArrayIndexOutOfBoundsException" as there are no values between the last three commas.

any ideas on how to split the data now?

Hi, sorry my bad. In my main code I have it only till e(4). Im getting indexoutofbounds at e(2), since there is no value after between conmas 2 and 3. Any fix for this? — murali krishna, Jul 29 '20 at 05:37

Suhas NM · Answer 1 · 2020-07-29T07:59:45.190

0

Your code is correct, but after splitting you are trying to access 6 elements instead of 5.

Change

.map(e ⇒ Row(e(0), e(1), e(2), e(3), e(4), e(5)))

to

.map(e ⇒ Row(e(0), e(1), e(2), e(3), e(4)))

UPDATE

By default, empty values are omitted when we do string split. That is the reason why your array has only 2 elements. To achieve what you intend to do, try this:

val Content = rddTest.map(_.toString().replace("[", "").replace("]", ""))
      .map(_.split(",",-1))
      .map(e ⇒ Row(e(0), e(1), e(2), e(3), e(4)))

observe the split function, using it that way will make sure all the fields are retained.

edited Jul 29 '20 at 07:59

answered Jul 29 '20 at 01:31

Suhas NM

960
7
10

Hi, sorry my bad. In my main code I have it only till e(4). Im getting indexoutofbounds at e(2), since there is no value after between conmas 2 and 3. Any fix for this? – murali krishna Jul 29 '20 at 04:36
I did not understand how you have initialized rdd ==> test.rdd will not work – Suhas NM Jul 29 '20 at 07:03
I have updated the answer. This should work for you – Suhas NM Jul 29 '20 at 11:52

score 0 · Answer 2 · answered Jul 29 '20 at 05:50

So dirty but replace several times.

val test = sc.parallelize(List("[hello,one,,,]"))

test.map(_.replace("[", "").replace("]", "").replaceAll(",", " , "))
    .map(_.split(",").map(_.replace(" ", "")))
    .toDF().show(false)

+------------------+
|value             |
+------------------+
|[hello, one, , , ]|
+------------------+

How to split RDD rows by commas when there is no value between them?

2 Answers2