15

So I have an S3 bucket of videos (several hundred), upon which I used ElasticTranscoder to transcode everything into a second, optimised bucket.

However, when I inspect my second bucket, there are 40-50 less objects, but I cannot figure out what they are (the directory structure is deeply nested etc).

How can I get the file diff of two buckets using aws s3api list-objects?

Perhaps there are files in the bucket which are not videos, which I somehow didn't know about.

j_d
  • 2,818
  • 9
  • 50
  • 91
  • Do you have a naming convention? Get all objects list from first bucket & 2nd bucket, massage the names & then get the diff?! – Mark Shehata Aug 04 '17 at 18:46

3 Answers3

27

You can use the sync command with the --dryrun option to compare instead of syncing.

aws s3 sync s3://bucket s3://bucket2 --dryrun

You can, of course, also use it to compare a local directory with a bucket.

aws s3 sync . s3://bucket2 --dryrun

George Hernandez
  • 433
  • 4
  • 6
23

Using Display only filenames:

aws s3 ls s3://bucket-1 --recursive | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ \t]*//' | sort > bucket_1_files
aws s3 ls s3://bucket-2 --recursive | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ \t]*//' | sort > bucket_2_files

diff bucket_1_files bucket_2_files
helloV
  • 50,176
  • 7
  • 137
  • 145
  • 1
    This approach is also great because it works even if your buckets are in different regions. – Andrew Guenther Oct 19 '22 at 17:07
  • 1
    This answer works for me. the `Total Objects:` indicated after running `aws s3 ls` are different for two bucket. Using this method, the diff is actually the folder was recognized as single line item in older bucket but in new bucket it was shown as `folder/object`. – rick Mar 14 '23 at 10:24
1

Inspired from @George comment

you can use this to extract paths list:

aws s3 sync s3://<main-bucket> s3://<second-bucket> --dryrun | awk 'match($3,"^(s3://[^/]+/)(.*)",a) {print a[2]}'

or for local paths

aws s3 sync <local-path> s3://darsak2.public --dryrun | awk 'match($3,"^(./)?(.*)",a) {print a[2]}'
Khaled AbuShqear
  • 1,230
  • 14
  • 24