I've write the data in the pyspark dataframe using the dataframe writer APIs. How can I change the name of the csv file generated?
Asked
Active
Viewed 1,439 times
1 Answers
2
It seems you are trying to get a single CSV file out of a Spark Dataframe, using the spark.write.csv() method. This will create a distributed file by default. I would recommend the following instead if you want a single file with a specific name.
df.toPandas().to_csv('/dbfs/path_of_your_file/filename.csv')
using Pandas to_csv arguments that fit your need.
Axel R.
- 1,141
- 7
- 22
-
how to parametrize the filename part? – Muhammad Waheed Jul 05 '22 at 15:24
-
You could use an fstring. Declare a variable ```filename = "name_of_my_file ``` and then ```df.toPandas().to_csv(f'/dbfs/path_of_your_file/{filename}.csv') ``` – Axel R. Jul 06 '22 at 13:11