pyspark - how to delete a local directory if it already exists

Question

i wish to delete a local directory if it already exists. below is my code:

import sys
import os
from pyspark import SparkContext
from pyspark import SparkConf

conf=SparkConf().setAppName('pyspark')
sc=SparkContext(conf=conf)

data=sc.textFile('file:///home/cloudera/Downloads/SAN_SALES_EXTRACT_TRANS_LEVEL_D0906.txt')
datamap=data.map(lambda x: ((str(x.split(',')[1]).strip(),int(x.split(",")[0])),float(x.split(",")[10])))
datagrouped=datamap.reduceByKey(lambda x,y: x+y)
if (os.path.exists("file:///home/cloudera/Downloads/store_perday_rev")):
        os.remove("file:///home/cloudera/Downloads/store_perday_rev")
else:
        datagrouped.sortByKey().saveAsTextFile("file:///home/cloudera/Downloads/store_perday_rev")
#for i in datagrouped.sortByKey().take(20):
#       print(i)

It doesn't delete the directory. What am i doing wrong?

Answer 1

Try os.rmdir() instead.

os.remove() only works for a file path, not for a directory.

Answer 2

You can try these options .

import os
os.rmdir("C:/test/delete/pydelete")

i am able to remove the folder.If you have the data in this folder then you need to call.

shutil.rmtree()

Answer 3

are your trying to remove the directory or the file?

if you are trying to remove the directory please refer to the following link:

How do I remove/delete a folder that is not empty with Python?

Also refer to the python docs: https://docs.python.org/2/library/os.html