How to copy file while preserving directory structure using AWS command line

Question

I have the following directory in my local computer:

dir1
  |
  |__ randomfile.jpg
  |__ dir2
        |
        |__ file1.txt
        |__ file2.txt
        |__ pict.png

What I want to do is to copy all the files with *.txt and preserving the subdirectory structure to Amazon S3 bucket. How can I do that?

At the end of the day in S3. We'd like to find this file and directory structure:

 dir1
      |
      |__ dir2
            |
            |__ file1.txt
            |__ file2.txt

With standard Unix command I can do this:

find . -name '*.txt' -exec cp --parents \{\} /target \;

But not sure how to do with with AWS command line.

In reality with have files with ~10Tb of size to transfer.

Possibly a duplicate of http://stackoverflow.com/questions/8659382/downloading-an-entire-s3-bucket — codeforester, Feb 06 '17 at 05:21

score 8 · Answer 1 · answered Feb 06 '17 at 05:17

8

Just use sync:

aws s3 sync src/ s3://mybucket --exclude "*" --include "*.txt"

answered Feb 06 '17 at 05:17

Ôrel

2

As an alternative to `sync` (which is intended for repeated syncing), you could use `aws s3 cp --recursive ...` which would do the same job. – John Rotenstein Feb 06 '17 at 09:40
@JohnRotenstein you could but exclude and include as of now work only with sync and not with cp or mv! – Karan Shah Feb 06 '17 at 12:05
@KaranShah that works with `cp` you need ` --recursive` – Ôrel Feb 06 '17 at 12:08
@Ôrel recursive will copy everything, it cannot exclude the random file as mentioned in the question – Karan Shah Feb 06 '17 at 12:09
My point is exclue and include are supported by cp, but here to keep structure you need sync I agree – Ôrel Feb 06 '17 at 12:10

1 Answers1