How to delete a Parquet file on Spark?

Question

I have saved a parquet file on Spark using DataFrame.saveAsParquet() command.

How can I delete/remove this file via python code?

score 5 · Accepted Answer · edited May 23 '17 at 12:25

5

This parquet "file" will actually be a directory. This answer shows how to delete a directory with files in it

import shutil
shutil.rmtree('/folder_name')

edited May 23 '17 at 12:25

Community

1
1

answered Jun 03 '16 at 14:48

David

11,245
3
41
46

It works without the forward slash. shutil.rmtree('folder_name') – guptashail Jun 05 '16 at 05:16
Note that this only works on the OS and is not applicable on HDFS, S3, or myriad other common locations for Parquet files being used in Spark. – bsplosion Dec 16 '19 at 21:56

score 0 · Answer 2 · answered May 15 '20 at 11:45

Since @bsplosion mentioned HDFS, here is how you could do it in a pySpark-script:

import subprocess

print("Deletion code:", subprocess.call(["hadoop", "fs", "-rm", "-r", "-skipTrash", "hdfs:/your/data/path"]))

# hadoop     - calls hadoop
# fs         - calls hadoops file system implementation
# -rm        - calls the remove command
# -r         - recursive removal in order to remove the entire directory
# -skipTrash - As it states: Skip the trash and directly remove everything

This returns Delection code: 0 if executed successfully, otherwise Delection code: -1. You can read more about hadoops -rm here in the docs.

How to delete a Parquet file on Spark?

2 Answers2