I am trying to delete the file 20221229_20230221-101756_Backtest_M.txt
given in hdfs path :
hdfs_path = '/dev/flux_entrant/depot/backtesting/'
To do it, I am using :
fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())
java_import(spark._jvm, 'org.apache.hadoop.fs.Path')
file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().endswith(suffix_file + '.txt')]
file_name_str = ''.join(file_name)
fs.delete(spark._jvm.org.apache.hadoop.fs.Path(f'{hdfs_path}/{file_name_str}'), True)
The Problem is that fs.delete remove the entire HDFS PATH, and this is bad because I wanna just removing the file and not the HDFS PATH
Any help please, because I did not find the command or the option that can do it ! Thanks !