how to recursively traverse through given HDFS directory and fetch the filename with path in pyspark

Asked Jun 17 '20 at 09:39

Active Jun 17 '20 at 09:50

Viewed 537 times

Structure of my directory in hdfs looks like below:

----Directory

 |----sub folder 1
          |--------file 1
          |---------file 2
 |                          
 |----sub folder 2
            |--------file 1
            |---------file 2
 |----sub folder 3
            |--------file 1
            |---------file 2
 |----file 1

i want to traverse recursively through the HDFS directory to retrieve all filename and its path.

edited Jun 17 '20 at 09:50

asked Jun 17 '20 at 09:39

Neetha Mary Paul

1

`org.apache.hadoop.fs.FileSystem#listFiles(, recursive=True)`? – Som Jun 17 '20 at 09:54
can you help with pyspark code to implement the above – Neetha Mary Paul Jun 18 '20 at 03:47
Please refer this answer- https://stackoverflow.com/a/45178292/4758823 – Som Jun 18 '20 at 04:05

how to recursively traverse through given HDFS directory and fetch the filename with path in pyspark

0 Answers0