Download all files from specific folder from Amazon S3 using boto3 Python

Question

As I am new to python.Need help in downloading all files from Specific pseudo-folder present inside S3 bucket.Below code starts downloading all files present inside bucket.How can I can achieve my goal.

import os
import errno
import boto3
import botocore


resource = boto3.resource('s3')
bucket = resource.Bucket('my-bucket')
client = boto3.client('s3')

def download_dir():
objList = client.list_objects(Bucket='my-bucket')['Contents']
for obj in objList:
    obj_Key = obj['Key']
    path,_destPath = os.path.split(obj_Key)
    print ("Downloading file :"+ obj_Key);
    client.download_file('my-bucket', obj_Key, _destPath)

download_dir()

Thanks in advance

What problems are you experiencing? You are likely to run into problems if there are objects in sub-directories, since the directories need to be created locally before being used. For some tips, see: [python - Boto3 to download all files from a S3 Bucket - Stack Overflow](https://stackoverflow.com/questions/31918960/boto3-to-download-all-files-from-a-s3-bucket/31929277) — John Rotenstein, Dec 12 '18 at 11:36
Hey thanks,for the above reference,since i was confused with the code,but your explaination for the above example cleared my doubts and its working. — Asif Iqbal, Dec 12 '18 at 11:51

score 0 · Answer 1 · answered Apr 07 '21 at 06:57

class Static_file_downloading:
    def __init__(self) -> None:
        self.client = boto3.client("s3")
        self.s3 = boto3.resource("s3")

    def getFiles(self):

        try:
            res = self.client.list_objects(
                Bucket="bucket-name",
                MaxKeys=5
            )
        except:
            print("No bucket found")
    
        result_set = []

        if "Contents" in res:
            for result in res["Contents"]:
                if result["Size"] > 0:
                    result_set.append(result)

        else:
            print("file not found")


        for i in range(len(result_set)):
            file_name = result_set[i]["Key"]
            print(f"downloading {file_name}")
            self.s3.meta.client.download_file(
                "bucket-name",
                file_name,
                path_to_download,
            )

score 0 · Answer 2 · edited Sep 07 '22 at 04:18

You’ll create an s3 resource and iterate over a for loop using objects.all() API.

for s3_object in my_bucket.objects.all():
    print("Download files list from s3 bucket..")
    print(s3_object.key)
    str = s3_object.key

s3.client.download_file() – API method to download file from your S3 buckets.

path, filename = os.path.split(s3_object.key)
my_bucket.download_file(s3_object.key, str)

For Full code : Click Here

Download all files from specific folder from Amazon S3 using boto3 Python

2 Answers2