Getting S3 objects' last modified datetimes with boto

Question

I'm writing a Python script that uploads files to S3 using boto librairy. I only want to upload changed files (which I can check by their "last modified" datetimes), but I can't find the Boto API endpoint to get the last modified date.

See http://mashupguide.net/1.0/html/ch16s07.xhtml towards the bottom. It implies that a bucket has keys and the keys have a `last_modified` attribute. Is that what you're looking for? Disclaimer: I've not used S3 :) — gfortune, Mar 13 '12 at 06:42

score 47 · Accepted Answer · answered Mar 13 '12 at 16:56

47

Here's a snippet of Python/boto code that will print the last_modified attribute of all keys in a bucket:

>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.lookup('mybucket')
>>> for key in bucket:
       print key.name, key.size, key.last_modified
index.html 13738 2012-03-13T03:54:07.000Z
markdown.css 5991 2012-03-06T18:32:43.000Z
>>>

answered Mar 13 '12 at 16:56

garnaat

44,310
7
123
103

3

For the record this is not perfect; the last_modified is a string, when what you probably want is a datetime – Chris R Mar 19 '12 at 19:10
14

If you want a DateTime, you can use boto.utils.parse_ts to parse the TZ string and return a DateTime object. – garnaat Mar 20 '12 at 15:42
5

oddly enough, the parse_ts function does not work with the format returned by key.last_modified i get time data 'Fri, 20 Jul 2012 16:57:27 GMT' does not match format '%Y-%m-%dT%H:%M:%S.%fZ' – Guillermo Siliceo Trueba Aug 02 '12 at 23:47
2

Just for future reference, this worked for me: time.strptime(key.last_modified, '%a, %d %b %Y %H:%M:%S %Z') – jdennison Nov 15 '13 at 18:33
1

It's worth noting that the format returned depends on how you got the key! Using boto 2.34.0, getting a key from `bucket.get_all_keys()` returns a string like `u'2015-03-16T14:02:50.000Z'`, whereas getting the same key though `bucket.get_key(name)` results in: `'Mon, 16 Mar 2015 14:02:50 GMT'`. In short: `bucket.get_key(bucket.get_all_keys()[0].name).last_modified == bucket.get_all_keys()[0].last_modified` is `False`. – staticfloat Apr 07 '15 at 04:12
1

That is true. It's because the S3 service sends the data in two different formats depending on whether you are listing the keys in a bucket or actually doing a HEAD or GET request on a particular key. – garnaat Apr 07 '15 at 04:23
thanks for your good explanation @garnaat – HootanHT Mar 14 '22 at 11:51

score 28 · Answer 2 · answered Mar 08 '19 at 10:25

Boto3 returns a datetime object for LastModified when you use the the (S3) Object python object:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.last_modified

You shouldn't need to perform any tortuous string manipulations.

To compare LastModified to today's date (Python3):

import boto3
from datetime import datetime, timezone

today = datetime.now(timezone.utc)

s3 = boto3.client('s3', region_name='eu-west-1')

objects = s3.list_objects(Bucket='my_bucket')

for o in objects["Contents"]:
    if o["LastModified"] == today:
        print(o["Key"])

You just need to be aware that LastModifed is timezone aware, so any date you compare with it must also be timezone aware, hence:

datetime.now(timezone.utc)

Is LastModifed UTC timezone aware always? – eduardosufan Aug 08 '22 at 15:02 — eduardosufan, Aug 08 '22 at 15:02

score 15 · Answer 3 · answered Jun 12 '20 at 04:03

For just one s3 object you can use boto client's head_object() method which is faster than list_objects_v2() for one object as less content is returned. The returned value is datetime similar to all boto responses and therefore easy to process.

head_object() method comes with other features around modification time of the object which can be leveraged without further calls after list_objects() result.

See this : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

import boto3

s3 = boto3.client('s3')
response = client.head_object(Bucket, Key)
datetime_value = response["LastModified"]

I think head_object is an apt call for metadata retrieval. – Amit Khetan Feb 09 '21 at 16:26 — Amit Khetan, Feb 09 '21 at 16:26

score 11 · Answer 4 · answered Apr 03 '14 at 07:45

11

this is working (tnx to jdennison from above):

after getting the key from s3:

import time
from time import mktime
from datetime import datetime

modified = time.strptime(key.last_modified, '%a, %d %b %Y %H:%M:%S %Z')

#convert to datetime
dt = datetime.fromtimestamp(mktime(modified))

answered Apr 03 '14 at 07:45

Eyal Ch

9,552
5
44
54

2

you answered the wrong issue - the issue is about getting date of s3 files, not about formatting it. – AdrianBR Oct 10 '17 at 11:43
time data u'2018-06-10T06:31:35.000Z' does not match format '%a, %d %b %Y %H:%M:%S %Z' – brown.2179 Jun 16 '18 at 17:07

score 4 · Answer 5 · answered May 07 '13 at 12:20

Convert the last_modified attribute to struct_time as given below

import time
for key in bucket.get_all_keys(): 
    time.strptime(key.last_modified[:19], "%Y-%m-%dT%H:%M:%S")

This will give a time.struct_time(tm_year, tm_mon, tm_mday, tm_hour, tm_min, tm_sec, tm_wday, tm_yday, tm_isdst) tuple for each key in the S3 bucket

score 3 · Answer 6 · answered Oct 04 '12 at 01:50

If you're using Django and django-storages, you can an unofficial API in the s3boto backend:

>>> from storages.backends.s3boto import _parse_datestring
>>> _parse_datestring("Fri, 20 Jul 2012 16:57:27 GMT")
datetime.datetime(2012, 7, 21, 2, 57, 27)

Unfortunately as of django-storages 1.1.5, this gives a naive datetime. You need to use django.utils.timezone to create an aware version:

>>> from django.utils import timezone
>>> naive = _parse_datestring("Fri, 20 Jul 2012 16:57:27 GMT")
>>> timezone.make_aware(naive, timezone.get_current_timezone())
datetime.datetime(2012, 7, 21, 2, 57, 27, tzinfo=<DstTzInfo 'Australia/Brisbane' EST+10:00:00 STD>)

score 2 · Answer 7 · answered May 07 '20 at 07:41

Using a Resource, you can get an iterator of all objects and then retrieve the last_modified attribute of an ObjectSummary.

import boto3

s3 = boto3.resource('s3')
bk = s3.Bucket(bucket_name)

[obj.last_modified for obj in bk.objects.all()][:10]

returns

[datetime.datetime(2020, 4, 17, 13, 23, 37, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 37, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 38, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 38, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 38, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 37, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 23, 37, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 17, 13, 20, 20, tzinfo=tzlocal()),
 datetime.datetime(2020, 4, 20, 8, 30, 2, tzinfo=tzlocal()),
 datetime.datetime(2020, 3, 26, 15, 33, 58, tzinfo=tzlocal())]

score 1 · Answer 8 · answered Oct 19 '21 at 14:03

1

You can get last object last modified date like that:

With `resource`

boto3.resource('s3').Object(<BUCKET_NAME>, <file_path>).last_modified

With `client`

boto3.client('s3').head_object(<BUCKET_NAME>, <file_path>)['LastModified']

answered Oct 19 '21 at 14:03

veben

19,637
14
60
80

score 0 · Answer 9 · answered Oct 09 '19 at 20:24

This is for recent s3 list_objectsv2. The boto3 client gives lastModifed in datetime.datetime format, and ways to convert it is as below
links: boto3 link
and aws s3 listobj

import datetime
from dateutil.tz import tzutc
# node s3 response '2019-06-17T18:42:57.000Z'
# python boto3 s3 response datetime.datetime(2019, 10, 1, 22, 41, 55, tzinfo=tzutc())
'''  {'ETag': '"c8ba0ad5003832f63690ea8ff9b66052"',
  'Key': 'SOMEFILE',
  'LastModified': datetime.datetime(2019, 10, 2, 18, 50, 47, tzinfo=tzutc()),
  'Size': 6390623,
  'StorageClass': 'STANDARD'}
'''
l = datetime.datetime(2019, 10, 1, 22, 41, 55, tzinfo=tzutc())
get_last_modified = int(l.strftime('%s'))
print(l)
print(get_last_modified)

score 0 · Answer 10 · answered Jan 05 '21 at 12:52

import boto3
from boto3.session import Session

session = Session(aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
my_bucket = s3.Bucket(BUCKET_NAME)

for obj in my_bucket.objects.all():
    print('{} | {}'.format(obj.key, obj.last_modified))

score 0 · Answer 11 · answered Mar 31 '22 at 01:36

You can try sorting the returned list of objects by LastModified key

import boto3

s3_client = boto3.client('s3')
s3_response = s3_client.list_objects(Bucket=BUCKET_NAME)
sorted_contents = sorted(s3_response['Contents'], key=lambda d: d['LastModified'], reverse=True)
sorted_contents[0].get('Key')

You can remove reverse=True flag in order to get the earliest modified object. You can also sort by Size of the objects or any other properties you want.

score 0 · Answer 12 · edited Jul 23 '22 at 03:55

Here is what I used for my lambda function...

Use s3.list_objects_v2(Bucket=Your_bucket_name) to list the objects then get the key LastModified from the contents.

import boto3
import json
import datetime  
   
s3 = boto3.client('s3')

def lambda_handler(event, context):
   bucket = Your-bucket-name
   try:
       listdata = s3.list_objects_v2(Bucket=bucket)
       contents = listdata['Contents'] if "Contents" in listdata else []
       for key in contents:
       lastmodified = str(key['LastModified'])
       print("lastmodified:", lastmodified)

Getting S3 objects' last modified datetimes with boto

12 Answers12

With `resource`

With `client`

Linked

Getting S3 objects' last modified datetimes with boto

12 Answers12

With resource

With client

Linked

With `resource`

With `client`