PySpark 2.4.4 toPandas fail with ValueError not enough values to unpack (expected 3, got 2)

Asked Jan 18 '20 at 05:55

Active Apr 20 '21 at 14:54

Viewed 881 times

On spark dataframe when I do a 'toPandas' i end up with this error:

pandas_df = spark_df.toPandas()

File "/opt/mapr/spark/spark-2.4.4/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 2122, in toPandas

File "/opt/mapr/spark/spark-2.4.4/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 2180, in _collectAsArrow

ValueError: not enough values to unpack (expected 3, got 2)

Spark version - 2.4.4 Pyarrow version - 0.14.1

The same works with Spark 2.3.1 and pyarrow 0.9.0.

Any help with solving this with spark 2.4.4?

edited Jun 20 '20 at 09:12

Community

asked Jan 18 '20 at 05:55

Manoj Srivatsav

also experiencing this issue. Hopefully someone will weigh in at some point – user2331566 Feb 12 '20 at 10:33
Did you ever figure this out? – rrs Dec 18 '20 at 14:34
possibly related to https://stackoverflow.com/questions/58269115/how-to-enable-apache-arrow-in-pyspark/58273294#58273294 – Alex Vorobiev Jun 29 '21 at 23:43

PySpark 2.4.4 toPandas fail with ValueError not enough values to unpack (expected 3, got 2)

0 Answers0