Downloading the latest file in an S3 bucket using AWS CLI?

Question

I have an S3 bucket that contains database backups. I am creating a script to download the latest backup (and eventually restore it somewhere else), but I'm not sure how to go about only grabbing the most recent file from a bucket.

Is it possible to copy only the most recent file from an S3 bucket to a local directory using AWS CLI tools?

What are you gonna use for scripting? SDK, CLI or something else? Also how are you naming the back files in S3 — , Jul 14 '16 at 22:11
What do you mean by most recent? dont the files have different names? — Shibashis, Jul 14 '16 at 22:25
@error2007s I'm using CLI. The files are named in a few different formats, but it typically contains a name and date component. I could potentially work on something to parse the dates but I would prefer not to if I can avoid it — Abe Miessler, Jul 14 '16 at 22:37
@Shibashis I mean the file that was most recently created. Yes - the files have different names. I am not using versioning. — Abe Miessler, Jul 14 '16 at 22:37
The accepted answer is a copy-and-paste duplicate of the accepted answer of the earlier question. See my comment below the accepted answer here. — Ali, Jun 22 '19 at 15:11
@AbeMiessler Were you able to setup automated download and restore process ? If yes, Can you please share the scripts you used. ? I would like to setup a automated job to download and restore from the latest backup file. — Krishna Chaitanya, Aug 10 '21 at 17:47

score 27 · Answer 1 · answered Sep 11 '17 at 21:26

27

And here is a bash script create based on @error2007s's answer. This script requires your aws profile and bucket name as variables, and downloads the latest object to your ~/Downloads folder:

#!/bin/sh
PROFILE=your_profile
BUCKET=your_bucket

OBJECT="$(aws s3 ls --profile $PROFILE $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}')"
aws s3 cp s3://$BUCKET/$OBJECT ~/Downloads/$OBJECT --profile $PROFILE

answered Sep 11 '17 at 21:26

JBaczuk

13,886
10
58
86

1

Worth noting that AWS now has Cloud Console so you can do this directly from the AWS Console. – jackofallcode May 12 '21 at 08:37
This might be more helpful by getting the more recent files (specify your date) from your bucket, then copy only those...using sort and tail perhaps to get the latest and list it (which could then be used for copying): ```aws s3api list-objects --bucket $BUCKET --query 'Contents[?LastModified>`2023-02-09`].{ Key: Key, LastModified: LastModified }' --output text | sort -k2 | tail -1``` [stack**overflow** Hat tip](https://stackoverflow.com/a/45433079/11972755). Hope this helps. – Michael Behrens Feb 11 '23 at 16:38
1

A better CLI method to get latest key per [updated answer](https://stackoverflow.com/a/58451831/11972755) is: ```aws s3api list-objects-v2 --bucket $BUCKET --query 'sort_by(Contents, &LastModified)[-1].Key' --output=text``` – Michael Behrens Feb 11 '23 at 17:13

suraj shetty · Answer 2 · 2020-02-04T17:26:47.903

FILE=`aws s3api list-objects-v2 --bucket "$BUCKET_NAME" --query 'reverse(sort_by(Contents[?contains(Key, \`$FILE_NAME_FILTER\`)], &LastModified))[:1].Key' --output=text`;aws s3 cp "s3://$BUCKET_NAME/$FILE" .

$BUCKET_NAME - is the bucket from which you want to download.

$FILE_NAME_FILTER - a string used as a filter for the name, which you want to match.

aws s3 cp " " - its in double-quotes because to also include files that have spaces in their names.

score 7 · Answer 3 · edited Jan 04 '22 at 16:12

The above solutions are using Bash. If one wants to do the same thing in PowerShell for downloading on Windows here is the script:

# This assumes AWS CLI exe is in your path.
$s3location = "s3://bucket-name"
$files = $(aws s3 ls $s3location --recursive | sort | select -last 3)
$dlPath = "C:\TEMP"

foreach ($s3FileInfo in $files) {
    $filename = $s3FileInfo.Split()[-1]
    $path = "${s3location}/${filename}"
    aws s3 cp $path $dlPath
    echo("Done downloading ${path} to ${dlPath}")
}

Downloading the latest file in an S3 bucket using AWS CLI?

3 Answers3

Linked