How to get rid of the row wrapper object in pyspark dataframes without using RDD api?

Question

I'm issuing the following SQL statement against a tempview

cloudantdata.createOrReplaceTempView("washingflat")

sqlDF = spark.sql("SELECT temperature FROM washingflat")
sqlDF.rdd.map(lambda row : row.temperature).collect()

I'm just interested in the plain (unwrapped) integer values. All I've tried so far with the dataframe API always returned Row-objects wrapping the values I'm interested in.

Is there a way to get the scalar contents without using the RDD api?

score 2 · Answer 1 · answered Nov 01 '16 at 19:59

2

You could manually put them in a list after collecting them as below

temps = []
rows = sqlDF.collect()
for r in rows:
    temps.append(r['temperature'])

answered Nov 01 '16 at 19:59

David

11,245
3
41
46

I'm really missing the map function on dataframes – Romeo Kienzler Nov 01 '16 at 20:07
`map` function converted `DataFrame` to `RDD`. There is no difference between old `df.map(...)` and new `df.rdd.map(...)`. `collect` converts to `RDD` first too. – Nov 01 '16 at 20:40
@RomeoKienzler Yea, Lost is correct. Check out http://stackoverflow.com/questions/39535447/attributeerror-dataframe-object-has-no-attribute-map/39536218#39536218 – David Nov 01 '16 at 20:57

score 1 · Answer 2 · 2016-11-01T20:08:42.853

1

Try:

>>> from itertools import chain
>>>
>>> chain.from_iterable(sqlDF.collect())

edited Nov 01 '16 at 20:08

answered Nov 01 '16 at 19:56

still uses rdd api :) – Romeo Kienzler Nov 01 '16 at 20:06
Sorry, didn't cut rdd part :) – Nov 01 '16 at 20:08

score 1 · Accepted Answer · answered Nov 01 '16 at 21:34

So given an input DataFrame

import numpy as np
import pandas as pd
test_df = pd.DataFrame({'Age': np.random.uniform(0,100, size = (100,)), 'City': 'LA'})
sqlContext.createDataFrame(test_df).registerTempTable('AgeTable')

There are two (primary) ways for extracting a value without using the Row abstraction. The first is to use the .toPandas() method of a DataFrame / SQL Query

print(sqlContext.sql("SELECT Age FROM AgeTable").toPandas()['Age'])

This returns a Pandas DataFrame / Series.

The second is to actually group the data inside of SQL and then extract it from a single Row object

al_qry = sqlContext.sql("SELECT City, COLLECT_SET(Age) as AgeList FROM AgeTable GROUP BY City")
al_qry.first()[0].AgeList

This returns a raw python list.

The more efficient way is with the toPandas method and this approach will likely be improved more in the future.

score 0 · Answer 4 · answered Jun 08 '18 at 10:36

0

temp_list = [str(i.temperature) for i in sqlDF.select("temperatue").collect()]

answered Jun 08 '18 at 10:36

braj

2,545
2
29
40

How to get rid of the row wrapper object in pyspark dataframes without using RDD api?

4 Answers4