Copy Files from AWS S3 to HDFS (Hadoop Distributed File System)

Asked Sep 16 '19 at 05:49

Active Sep 16 '19 at 08:55

Viewed 579 times

I'm trying to copy AVRO files from AWS S3 bucket to HDFS using the following Scala code:

val avroDF  = sparkSession.read.format("com.databricks.spark.avro").load("s3a://"+s3Location+"/")
avroDF.write.format("com.databricks.spark.avro").mode(SaveMode.Append).save(filePath)

The files when being copied to HDFS, part files are getting saved like (part-0001.avro), how to save the file with the same file name as it exists in AWS S3 bucket?

edited Sep 16 '19 at 08:55

Abdullah Khawer

4,461
4
29
66

asked Sep 16 '19 at 05:49

Swathi

What is the name of the file `part-0001.avro` on S3? – Abdullah Khawer Sep 16 '19 at 05:51
Have you tried using s3distcp with Spark? Check this out: https://stackoverflow.com/questions/42095983/adding-s3distcp-to-pyspark – Abdullah Khawer Sep 16 '19 at 06:00

Copy Files from AWS S3 to HDFS (Hadoop Distributed File System)

0 Answers0