Spark - Write DataFrame with custom file name

Question

I have a Spark (2.4) DataFrame that I want to write as a Pipe separated file. It should be pretty straightforward like so

val myDF = spark.table("mySchema.myTable")
myDF.coalesce(1).write.format("csv").options("header", "true").options("delimiter", "|").save("/tmp/myDF")

I get a part-*.csv file in /tmp/myDF.

So far, so good. But I actually want the file name to be something specific, e.g. /tmp/myDF.csv

But giving this String in save will just create a dir called myDF.csv and create the part*.csv file in there.

Is there a way to write the DataFrame with a specific name?

Answer 1

You can't do that with Spark

You can rename the file later accessing the fileSystem

val directory = new File(/tmp/myDF)

if (directory.exists && directory.isDirectory) {
   val file = directory.listFiles.filter(_.getName.endsWith(".csv")).head
   file.renameTo("myDF.csv")
}