ValueError: setting an array element with a sequence when plotting line chart using pyspark in zeppelin

Question

Currently I have this dataframe(df):

    +-------------------+--------+
    |       datetime_new|pkts_new|
    +-------------------+--------+
    |2018-01-04 00:00:00|    47.0|
    |2018-01-04 00:00:00|     1.0|
    |2018-01-04 00:00:00|     3.0|
    |2018-01-04 00:00:00|    40.0|
    |2018-01-04 00:01:00|     1.0|
    |2018-01-04 00:01:00|     1.0|
    |2018-01-04 00:01:00|     1.0|
    |2018-01-04 00:01:00|    49.0|
    |2018-01-04 00:01:00|    33.0|
    |2018-01-04 00:01:00|     2.0|
    +-------------------+--------+

`DataFrame[datetime_new: timestamp, pkts_new: double]`

type(t): <class 'pyspark.sql.dataframe.DataFrame'>

I want to plot a line chart where the x axis is datetime_new, and the y axis is pkts_new. However, when i used this command: plt.plot(df.select('datetime_new'),df.select('pkts_new')) , I will get the error message:ValueError: setting an array element with a sequence.

How do I plot a line chart for ?

score 0 · Answer 1 · answered Dec 20 '18 at 07:08

0

Convert array into list using list(myarray)

import matplotlib.pyplot as plt    
plt.plot(data_time,time)
plt.show()

answered Dec 20 '18 at 07:08

raunak rathi

95
1
9

If i were to type `plt.plot(list(df.select("datetime_new")),list(df.select("pkts_new")))`, the error will still be the same – R_abcdefg Dec 20 '18 at 07:21
convert the array in list at prior stage during plotting it will not convert to list, and plotting needs list for axis – raunak rathi Dec 20 '18 at 07:24
`t_dt = list(df.select("datetime_new")) t_pkt = list(df.select("pkts_new")) plt.plot(t_dt , t_pkt) plt.show()` wil still give me the same error message. can u show me an example please? Thanks – R_abcdefg Dec 20 '18 at 07:27
https://stackoverflow.com/a/22341390/4683899 refer this link for conversion – raunak rathi Dec 20 '18 at 07:28
plt.plot(df['datetime_new'].tolist(),df['pkts_new'].tolist()) I guess this will help – raunak rathi Dec 20 '18 at 07:57
TypeError: 'Column' object is not callable, because my dataframe is not pandas. is pyspark sql – R_abcdefg Dec 20 '18 at 08:01
https://stackoverflow.com/a/49123695/4683899 this is conversion of pysark into list – raunak rathi Dec 20 '18 at 08:04

ValueError: setting an array element with a sequence when plotting line chart using pyspark in zeppelin

1 Answers1