How can find size of each Row in Apache spark sql dataframe and discrad the rows having size more than a threshold size in Kilobyte

Question

I am new in apache spark sql in scala.

How can find size of each Row in Apache spark sql dataframe and discrad the rows having size more than a threshold size in Kilobyte. I am looking for scala solution.

score 7 · Accepted Answer · edited May 23 '17 at 12:24

This is actually kind of a tricky problem. Spark SQL uses columnar data Storage so thinking of individual row sizes isn't super natural. We can of course call .rdd on from there you can filter the resulting RDD using the techniques as from Calculate size of Object in Java to determine the object size, and then you can take your RDD of Rows and convert it back to a DataFrame using your SQLContext.

How can find size of each Row in Apache spark sql dataframe and discrad the rows having size more than a threshold size in Kilobyte

1 Answers1

Linked