10

I have a data frame with three columns and I am trying to do a line plot using Seaborn library but it throws me an error saying that 'DataFrame' object has no attribute 'get'. Here is my test data frame

Age variable    value
31  Overall 69.76751118
31  Potential   69.76751118
31  Growth  0
34  Overall 68.91176471
34  Potential   68.91176471
34  Growth  0
28  Overall 69.05803996
28  Potential   69.05803996
28  Growth  0.24643197

This is what I am trying to do using the seaborn line plot after reading in the csv file

test = spark.read.csv("test.csv", inferSchema=True, header=True)
sns.lineplot(x = "Age", y = "value", hue = "variable", data = test)

And the error that I get is this

AttributeError: 'DataFrame' object has no attribute 'get'

However when I convert the data frame to Pandas data frame and use exactly the same seaborn code it works

test_df = test.toPandas()
sns.lineplot(x = "Age", y = "value", hue = "variable", data = test_df)

enter image description here

Am I doing anything wrong with Spark Data frames.

upendra
  • 2,141
  • 9
  • 39
  • 64
  • Not possible- you need to [convert to a pandas DataFrame](https://stackoverflow.com/questions/41826553/convert-between-spark-sql-dataframe-and-pandas-dataframe) first, which is going to be expensive. – pault Nov 01 '18 at 16:17
  • Is there an alternative other than converting to Pandas Dataframe? – upendra Nov 01 '18 at 18:15
  • 2
    Any solution will require the data to be on your local machine, which involves a `collect` type operation. – pault Nov 01 '18 at 18:18
  • Ah I see. I will give it a try and see if that works. Thanks.. – upendra Nov 01 '18 at 18:42

1 Answers1

10

A spark dataframe and a pandas dataframe, despite sharing a lot of the same functionalities, differ on where and how they allocate data.

This step is correct:

test_df = test.toPandas()

You will always need to collect the data before you can use it to plot with seaborn (or even matplotlib)

Maviles
  • 3,209
  • 2
  • 25
  • 39
  • 8
    Be very careful while doing this as this will pull all the data to driver node. On large dataset it might cause OOM. – Nandeesh Jun 28 '19 at 12:00
  • @Nandeesh Exactly! This makes distributed spark paradigm pointless...what's the solution? – Pablo Boswell Apr 24 '22 at 20:55
  • 1
    @PabloBoswell, the problem is that the data reduction generally is done inside the plotting library. So the solution is, instead of downloading millions of rows of data and plotting a histogram, you do the data reduction in spark and create the exactly same view using a bar plot and downloading only 10 rows of data from spark. Despite simple, I never found a python package dedicated to it, ready for histograms, density, etc. – Maviles May 09 '22 at 17:34