1

I can run this command for HDFS:

hadoop fs -ls /user/hive/warehouse/databasename.db/tablename

How to write command in Spark to show all files under specific folder in HDFS?

Thanks.

Joe
  • 11,983
  • 31
  • 109
  • 183
  • 2
    This might be a possible copy of https://stackoverflow.com/questions/27023766/spark-iterate-hdfs-directory **FileSystem** has the majority of HDFS commands available so it might fit your needs. – kzivic Jun 10 '19 at 15:06

1 Answers1

1

OK, the below scala code just give you a function to print all the hdfs files under a parent path.You can improve it according to your needs.

  def getAllPaths(parentPath:String, fs: FileSystem) = {
    val fileStatus = fs.listStatus(new Path(parentPath))
    for( file<- fileStatus) {
      println(file.getPath.toString)
    }
  }
Shawn.X
  • 1,323
  • 6
  • 15