S3 listing all files in subfolder in a bucket

Question

I have an s3 bucket, with files under a folder structure like folder1/folder2 I want to list the files under the folder structure only and iterate through the files in a Sagemaker Jupyter notebook.

How can I achieve this? I tried the instructions in Listing contents of a bucket with boto3 but was able to list only at the top level recursively. But I want to list only at the folder level.

I also tried the below code snippet

import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucketname/folder1/folder2')
for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object)

and got the below error

ParamValidationError: Parameter validation failed:
Invalid bucket name...

Using Python 3.9 currently. Thanks!

Answer 1

A few issues here:

1. bucketname is the bucket name
2. folder1/folder2/ is the key prefix
3. you need to filter the list, not get all objects

Try:

import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('bucketname')
for object_summary in bucket.objects.filter(Prefix='folder1/folder2/'):
    print(object_summary)

That will result in a list of ObjectSummary values being printed, for example:

s3.ObjectSummary(bucket_name='bucketname', key='folder1/folder2/')
s3.ObjectSummary(bucket_name='bucketname', key='folder1/folder2/abc.csv')
s3.ObjectSummary(bucket_name='bucketname', key='folder1/folder2/def.csv')
s3.ObjectSummary(bucket_name='bucketname', key='folder1/folder2/xyz.png')
s3.ObjectSummary(bucket_name='bucketname', key='folder1/folder2/folder3/')

Note that it will include all objects at the folder1/folder2/ level, regardless of their file extension suffix, and it will potentially include an indication of the folder itself (folder1/folder2/) and any logical sub-folders such as folder1/folder2/folder3/.

You can retrieve the objects from an object summary as follows:

for object_summary in bucket.objects.filter(Prefix="folder1/folder2/"):
    print(object_summary.Object().key)

That will result in a list of Object keys being printed, for example:

folder1/folder2/
folder1/folder2/abc.csv
folder1/folder2/def.csv
folder1/folder2/xyz.png
folder1/folder2/folder3/

You can filter these to get just CSVs, as needed, for example:

summaries = bucket.objects.filter(Prefix="folder1/folder2/")
csvs = [x for x in summaries if x.Object().key.endswith(".csv")]

for objectsummary in csvs:
    print(objectsummary.Object().key)

That will result in:

folder1/folder2/abc.csv
folder1/folder2/def.csv

And you can split out the actual filename, as follows:

for objectsummary in csvs:
    print(objectsummary.Object().key.split("/")[-1])

That will result in:

abc.csv
def.csv