0

I wrote the following Python script to download ALL files within an S3 Bucket into my current directory: 

import boto3
import botocore
import os

from boto3.session import Session

ACCESS_KEY='AWS_IAM_AccessKey'
SECRET_KEY='AWS_IAM_SecretKey'

session = Session(aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
myBucket = s3.Bucket('S3_bucketName')

for object in thamesBucket.objects.all():
      myBucket.download_file(object.key, os.path.join(os.curdir, os.path.basename(object.key)))

I'd like to further enhance this script to only pull down S3 files generated within the last 24 hours (using the Last Modified column value?) as opposed to all of them.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
QAE
  • 13
  • 1
  • 5
  • 1
    If your goal is to obtain files that have been added since the last download, you could instead consider using the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/) `aws s3 sync` command. It will only copy files that have been changed/modified since the last `sync`. – John Rotenstein Jul 26 '18 at 23:25
  • Side-note: It is recommended to _never_ put AWS credentials within source code. Instead, store them in the `~/.aws/credentials` file. (An easy way to create this is with the `aws configure` command.) If you are running the code on an Amazon EC2 instance, you should instead assign an IAM Role to the instance and credentials will be automatically provided without requiring a credentials file. – John Rotenstein Jul 26 '18 at 23:27

1 Answers1

1

This seems to work:

from datetime import datetime, timedelta
from dateutil.tz import tzutc, UTC
import boto3

s3 = boto3.resource('s3', region_name='YOUR-REGION')
bucket = s3.Bucket('YOUR-BUCKET')

for object in bucket.objects.all():
    if object.last_modified > datetime.now(tzutc()) - timedelta(hours = 24):
        <download code here>
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • wow...your code works like a charm.. thanks a lot you have saved a lot of time, I was not able to find time delta but thank you so much. I will be following you from now. – Dipen Chawla Feb 23 '20 at 15:20