1

I am writing spark output to an external system that does not like file extensions (I know, I know, don't start).

Something like:

   df.write.partitionBy("date").parquet(some_path)

creates files like: some/path/date=2021-01-01/part-00000-77dd02e8-1a67-4f0d-9c07-b55b4f2e5efc-c000.snappy.parquet

And that makes that external system unhappy. I am looking for a way to tell spark to write that file without extension. I know I can just rename it afterwards, but that seems ... stupid (and there's a lot of files) :/

Is there some option I could use to tell spark to just write it the way I want?

Dima
  • 39,570
  • 6
  • 44
  • 70
  • 3
    seems like you can't like the answer [here](https://stackoverflow.com/a/36108367/9477843) – AdibP Aug 18 '21 at 22:40
  • Does this answer your question? [Change output filename prefix for DataFrame.write()](https://stackoverflow.com/questions/36107581/change-output-filename-prefix-for-dataframe-write) – thebluephantom Aug 19 '21 at 09:52

1 Answers1

-1

df.write.partitionBy("date").parquet(some_path_ends_with_file_name)

Karthick Ak
  • 1