s3distcp - takes long time to copy large number of small files from one bucket to another

Asked May 28 '17 at 09:19

Active Mar 13 '18 at 13:39

Viewed 581 times

I need to copy large number of small files from one S3 bucket to another. I'm using S3-Dist-Cp command provided by AWS.

s3-dist-cp --src=s3://some-bucket/ --dest=s3://another-bucket/ --groupBy=<some-pattern> --targetSize=<size> --deleteOnSuccess

Now, the problem with this command is that it takes forever to copy all small files and merge them.

Note - Source bucket is being written continuously with new files by some other job and I think s3-dist-cp never catches with last file.

Is there any workaround for this solution? destination bucket will be used by Spark job to process these files.

asked May 28 '17 at 09:19

hlagvankar

Did you get any solution for this? – Cesar A. Mostacero Dec 18 '19 at 23:04
I am facing the same performance issue, did you find a way to fix it ? – Fabien Roussel Jun 06 '20 at 22:10

0 Answers0