Boto: How to use s3 as a resource or a client, but not both?

Question

This is my complete newb intro to Boto and AWS. At the moment my only goal is to be able to access an external agency's S3 bucket, so I want to understand how to use them in particular. This minimal code does what I want it to do, but I haven't figured out how to use this by only declaring s3r as a resource and avoiding having to use s3 as a client. It seems like it would be better to access the bucket from the S3 resource and then work exclusively with the bucket, i.e., bucket.new_key('testdir/') or bucket.put_object(Key=('testdir/)). Is this possible, or is there alternately a good reason to reframe how I'm approaching this? Thanks!

import boto3

bucket_name = 'my-bucket-name'
region_name = 'my-region-name'

print('Acquiring s3 service')
s3 = boto3.client('s3', region_name=region_name)
s3r = boto3.resource('s3', region_name=region_name)OB

print('Accessing bucket')
bucket = s3r.Bucket(bucket_name)

print('Emptying bucket')
bucket.objects.all().delete()

print('Uploading folder structures')
s3.put_object(Bucket=bucket_name, Key=('testdir/'))
s3.put_object(Bucket=bucket_name, Key=('testdir/subdir1/'))
s3.put_object(Bucket=bucket_name, Key=('testdir/subdir2/'))

What do you mean by "s3 as a client". This statement is confusing. — smac2020, Jun 24 '22 at 17:32
@smac2020 They mean just using the `boto3.client`, which is a lower lever of abstraction than `boto3.resource`. — wim, Jun 24 '22 at 17:36
Agreed.....S3 is a service, specifically 'Simple Storage Service'.....not a client. Can you clarify? — Mark J. Bobak, Jun 24 '22 at 17:37
I'll update it in the question, but i mean specifically accessing s3 as a module via `s3c = boto3.client('s3')` vs `s3r = boto3.resource('s3')`. I would hope not to have to declare both of these, and the tutorial I've been using as a resource uses the `s3r` approach, but I can't seem to do everything with just s3r. I'm also just not clear on the difference between the two objects. — Frank Harris, Jun 24 '22 at 17:40

Anon Coward · Accepted Answer · 2023-08-02T19:04:11.827

The boto3 API provides both a 'client' and 'resource' object model for most of the AWS APIs. The documentation has this to say on the difference (with a caveat I'll mention later):

Resources represent an object-oriented interface to Amazon Web Services (AWS). They provide a higher-level abstraction than the raw, low-level calls made by service clients

In other words, the 'client' APIs are a fairly one to one wrapper over the underlying AWS REST calls. The 'resource' API calls are meant to be easier to use, and they provide some "quality of life" improvements that make writing code quicker. Which one to use largely comes down to a coding style preference. For the most part what you can accomplish with 'client' calls can also be accomplished with 'resource' calls. Not always, though. Certainly, for your example, it's possible in either case:

s3 = boto3.client('s3')

# List all of the objects in a bucket, note that since we're fairly
# close to the underlying REST API with the client interface, we need
# to worry about paginating the list objects
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket=bucket_name):
    for cur in page.get('Contents', []):
        # And delete each object in turn
        s3.delete_object(Bucket=bucket_name, Key=cur['Key'])

# Create a zero-byte object to represent the folder
s3.put_object(Bucket=bucket_name, Key='testdir/')

The same work can be accomplished with the resource interface

s3r = boto3.resource('s3')

# Same idea with resource
bucket = s3r.Bucket(bucket_name)
# Paginating, and calling delete on each object in turn is handled
# behind the scenes by all() and delete() in turn
bucket.objects.all().delete()
# Creating the object, again make a zero-byte object to mimic creating
# a folder as the S3 Web UI does
bucket.put_object(Key='testdir/')

There is a caveat in the documentation, however:

The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3’s lifecycle. Customers can find access to newer service features through the client interface.

It comes down to personal preferences. I personally prefer using the client interface, since it makes it easier to understand and track which underlying API calls are being made as well as provides new AWS functionality, but it's really up to you.

Boto: How to use s3 as a resource or a client, but not both?

1 Answers1