0

I am trying to delete the file 20221229_20230221-101756_Backtest_M.txt given in hdfs path :

hdfs_path = '/dev/flux_entrant/depot/backtesting/'

To do it, I am using :

fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())
java_import(spark._jvm, 'org.apache.hadoop.fs.Path')
file_name          = [file.getPath().getName() for file in list_status if file.getPath().getName().endswith(suffix_file + '.txt')]
file_name_str      = ''.join(file_name)
fs.delete(spark._jvm.org.apache.hadoop.fs.Path(f'{hdfs_path}/{file_name_str}'), True)

The Problem is that fs.delete remove the entire HDFS PATH, and this is bad because I wanna just removing the file and not the HDFS PATH

Any help please, because I did not find the command or the option that can do it ! Thanks !

user8810618
  • 115
  • 11

0 Answers0