How to save file in Feather format\storage from Spark?

Question

Is it possible to export data-frame from Apache Spark to feather (https://github.com/wesm/feather) file?

score 2 · Answer 1 · answered Oct 17 '18 at 10:54

2

Not sure, you can do it directly, but you can transform first the Spark Dataframe (on pyspark) to a pandas and store it the to Feather:

pandas_df = spark_df.toPandas()

feather.write_feather(pandas_df, 'example_feather')

But I afraid, this will have an impact on the performance.

answered Oct 17 '18 at 10:54

Christoph Haene

4

Yeah, it works only if your data-frame fits into memory. So, not exactly what I was looking for. – Dmitry Petrov Oct 22 '18 at 00:25

1 Answers1