I have a large Spark dataframe df which I need to write on local disk.
df.toPandas().to_csv() does not work since this consume
too much memory.
How to split the dataframe and write down into a single file ?
Asked
Active
Viewed 67 times
0
OneCricketeer
- 179,855
- 19
- 132
- 245
quantCode
- 495
- 1
- 5
- 12
-
You should consider filtering and projecting the data that you actually need. How large are you talking about? – OneCricketeer Feb 06 '18 at 14:16
-
yes, thinking of split by slice : df[slicei,:] . How can I do it ? – quantCode Feb 06 '18 at 22:56
-
Well, Spark dataframes aren't the same as Python lists, so I don't think the slicing notation works in that context – OneCricketeer Feb 07 '18 at 00:38