30

I need to send backup files of ~2TB to S3. I guess the most hassle-free option would be Linux scp command (have difficulty with s3cmd and don't want an overkill java/RoR to do so).

However I am not sure whether it is possible: How to use S3's private and public keys with scp, and don't know what would be my destination IP/url/path?

I appreciate your hints.

Community
  • 1
  • 1
qliq
  • 11,695
  • 15
  • 54
  • 66
  • p.s. After moving to the latest version of s3cmd and eliminating --private-ack from the put command my problem with s3cmd seem to be resolved. – qliq Sep 11 '11 at 20:50
  • I guess you can mount S3 within EC2 (http://michaelaldridge.info/post/12086788604/mounting-s3-within-an-ec2-instance) and do the SCP! – cbacelar Mar 19 '13 at 15:17
  • As around ~Nov 2018, with [AWS transfer family](https://aws.amazon.com/aws-transfer-family/) you can now add a FTP or SFTP server for your s3 buckets and folders. – tread Sep 23 '22 at 07:59

7 Answers7

52

As of 2015, SCP/SSH is not supported (and probably never will be for the reasons mentioned in the other answers).

Official AWS tools for copying files to/from S3

  1. command line tool (pip3 install awscli) - note credentials need to be specified, I prefer via environment variables rather than a file: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.

    aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "*" --include "*.jpg"

    and an rsync-like command:

    aws s3 sync . s3://mybucket

  2. Web interface:

Non-AWS methods

Any other solutions depend on third-party executables (e.g. botosync, jungledisk...) which can be great as long as they are supported. But third party tools come and go as years go by and your scripts will have a shorter shelf life.

EDIT: Actually, AWS CLI is based on botocore:

https://github.com/boto/botocore

So botosync deserves a bit more respect as an elder statesman than I perhaps gave it.

Sridhar Sarnobat
  • 25,183
  • 12
  • 93
  • 106
  • I wonder if there's a way to ssh to a server then use an AWS shell instead of a traditional shell that exposes the raw file system (which AWS S3 intentionally abstracts away from). – Sridhar Sarnobat Jun 30 '17 at 01:36
  • 2
    Just a note that on ubuntu you can `apt-get install awscli` this may be preferable for some to using `pip3 install awscli` – Philip Couling Jun 07 '18 at 10:22
11

Here's just the thing for this, boto-rsync. From any Linux box, install boto-rsync and then use this to transfer /local/path/ to your_bucket/remote/path/:

boto-rsync -a your_access_key -s your_secret_key /local/path/ s3://your_bucket/remote/path/

The paths can also be files.

For a S3-compatible provider other than AWS, use --endpoint:

boto-rsync -a your_access_key -s your_secret_key --endpoint some.provider.com /local/path/ s3://your_bucket/remote/path/
datasn.io
  • 12,564
  • 28
  • 113
  • 154
8

You can't SCP.

The quickest way, if you don't mind spending money, is probably just to send it to them on a disk and they'll put it up there for you. See their Import/Export service.

El Yobo
  • 14,823
  • 5
  • 60
  • 78
  • Thanks, but that's over-overkill :) – qliq Sep 07 '11 at 04:30
  • 2
    Well, you do have two TB, it's going to take a while even on very fast networks :) You can't SCP because it's not like EC2, there is no machine running an SSH server for you to SCP to. You can upload using their APIs, try the info at http://stackoverflow.com/questions/667478/how-do-i-bulk-upload-to-s3. – El Yobo Sep 07 '11 at 04:35
  • 1
    'A List of Amazon S3 Backup Tools' may be helpful to avoid paying Amazon: http://jeremy.zawodny.com/blog/archives/007641.html – qliq Sep 07 '11 at 04:36
  • 1
    My server can upload at 5-10MB/s. In addition, I do that using crontab, so literally don't need to 'wait' for backup upload to complete. That's why I'm looking for a command-line solution. – qliq Sep 07 '11 at 04:49
  • http://www.jets3t.org claims to have what you need, "Synchronize: A command-line application for synchronizing directories on your computer with an Amazon S3 or Google Storage account. Ideal for performing back-ups or synchronizing files between different computers.". – El Yobo Sep 07 '11 at 04:56
7

Here you go,

scp USER@REMOTE_IP:/FILE_PATH >(aws s3 cp - s3://BUCKET/SAVE_FILE_AS_THIS_NAME)
GypsyCosmonaut
  • 661
  • 12
  • 17
4

Why don't you scp it to an EBS volume and then use s3cmd from there? As long as your EBS volume and s3 bucket are in the same region, you'll only be charged for inbound data charges once (from your network to the EBS volume)

I've found that once within the s3 network, s3cmd is much more reliable and the data transfer rate is far higher than direct to s3.

robotrobot
  • 193
  • 1
  • 8
1

There is an amazing tool called Dragon Disk. It works as a sync tool even and not just as plain scp.

http://www.s3-client.com/

The Guide to setup the amazon s3 is provided here and after setting it up you can either copy paste the files from your local machine to s3 or setup an automatic sync. The User Interface is very similiar to WinSCP or Filezilla.

Koustav Ray
  • 1,112
  • 13
  • 26
-3

for our AWS backups we use a combination of duplicity and trickle duplicity for rsync and encryption and trickle to limit the upload speed

ssmithstone
  • 1,219
  • 1
  • 10
  • 21