Is there anyway to give a custom name to a csv when it is generated from pyspark dataframe?

Asked Aug 21 '23 at 21:26

Active Aug 21 '23 at 23:05

Viewed 41 times

I need to give a custom name to a csv file generated from a pyspark dataframe. Downstream needs csv file with a custom name like "extraction_on_YYYYMMDD.csv" but pyspark always generate names like "part.......csv"

So far using pyspark I could not give a custom name. So I am writing data from pyspark at a location. Then reading that csv back using pandas and writing it back from pandas dataframe with custom name.

Please let me know if there is any better way to have a custom name from pyspark dataframe while writing it to csv.

edited Aug 21 '23 at 23:05

Abdennacer Lachiheb

4,388
7
30
61

asked Aug 21 '23 at 21:26

Praveen Choudhary

1

Can you not simply rename the file instead of reading it with pandas and writing another file? – mkrieger1 Aug 21 '23 at 21:30
Here's a similar question: https://stackoverflow.com/a/75929454/6325994 – Abdennacer Lachiheb Aug 21 '23 at 23:04
as @mkrieger1 suggested, if you really need to rename the files, you could probably use the `glob` library to read in all of the files, and then rename them – Derek O Aug 22 '23 at 15:39
This answer could help you out. https://stackoverflow.com/a/62434562/3238085 – user238607 Aug 25 '23 at 12:18

Is there anyway to give a custom name to a csv when it is generated from pyspark dataframe?

0 Answers0