How can I retrieve a folder from S3 into an AWS SageMaker notebook

Question

I have a folder with several files corresponding to checkpoints of a RL model trained using RLLIB. I want to make an analysis of the checkpoints in a way that I need to pass a certain folder as an argument, e.g., analysis_function(folder_path). I have to run this line on a SageMaker notebook. I have seen that there are some questions on SO about how to retrieve files from s3, such as this one. However; how can I retrieve a whole folder?

Answer 1

To read the whole folder, you will just have to list all files in the folder and loop through them. You could either do something like -

import boto3
s3_res = boto3.resource("s3")
my_bucket = s3.Bucket("<your-bucket-name>")

for object in my_bucket.objects.filter(Prefix="<your-prefix>")
    # your code goes here

Or, simply download the files to your local storage and loop them as you see fit (copy reference)-

!aws s3 cp s3://bucket/prefix/ . --recursive