How do I download a folder from AWS S3 with CloudPathLib from a public bucket?

Question

I'm trying to download a folder from a public AWS S3 bucket using the Python library cloudpathlib. My code looks like this:

from cloudpathlib import CloudPath
path = r"C:\some\path\to\folder"
url = "s3://some-example-bucket/folder/"

cloud_path = CloudPath(url)
cloud_path.download_to(path)

Really straight forward. To my knowledge, this should work, because the bucket is public:

Here is the bucket policy (nabbed from the AWS S3 tutorial):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::cozy-auto-texture-sd-repo/*"
        }
    ]
}

When I run the Python code to install the folder stable-diffusion-v1-4 the following error message appears:

  File "E:\Cozy-Auto-Texture-Files\venv\lib\site-packages\botocore\auth.py", line 418, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

My question is why is this happening? My bucket shouldn't require credentials since it's public. Is there something I'm missing with the buckets permissions or is it a Python code thing?

Thank you for reading and I appreciate the help!

Edit:

I've tried again with this method:

BUCKET_NAME = 'cozy-auto-texture-sd-repo'  # replace with your bucket name
KEY = 'stable-diffusion-v1-4'

s3 = boto3.resource('s3')

try:
    s3.Bucket(BUCKET_NAME).download_file(KEY, sd_path)
except botocore.exceptions.ClientError as e:
    if e.response['Error']['Code'] == "404":
        print("The object does not exist.")
    else:
        raise

However the same error message botocore.exceptions.NoCredentialsError: Unable to locate credentials appears, leading me to believe there is something wrong with my S3 bucket setup.

@Damienknight Possibly? I'm running this in a Venv so maybe there is something there that's the issue. Do you mean `path` or the path `cloudpathlib` was installed to? — Torrin worx, Sep 13 '22 at 19:24
I mean your 'path'. Try taking the s3 path out of the equation using cloudpathlib.local for testing — Damienknight, Sep 13 '22 at 19:56
@Damienknight I've tried again with another method (see the edit above) and I'm still getting the same error. There isn't anything wrong with the save path that I can see since its just going to an empty drive. — Torrin worx, Sep 13 '22 at 20:34
This is answered in the docs: https://cloudpathlib.drivendata.org/stable/authentication/#accessing-public-s3-buckets-without-credentials — hume, Jul 28 '23 at 18:15

score 0 · Answer 1 · answered Sep 14 '22 at 20:16

Assuming you are running this script locally, you may need to first set up your AWS credentials on your local machine.

See this existing answer:

It worked with an ~/.aws/credentials file containing:

[default]
aws_access_key_id=XXXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYYYYYYYYYYYYYYYYY

Using aws configure also works if you have aws-cli installed.

score 0 · Accepted Answer · answered Sep 17 '22 at 13:35

0

It turns out that AWS S3 doesn't allow you to download a folder from a public bucket. I got around this by simply zipping the folder I was trying to distribute and unzipping it in my application when the download was complete.

answered Sep 17 '22 at 13:35

Torrin worx

164
7

score 0 · Answer 3 · answered Mar 10 '23 at 07:05

if you have aws cli you could do something using subprocess.call.

from pathlib import Path
path = Path(r"C:\some\path\to\folder")
import subprocess
url = CloudPath("s3://some-example-bucket/folder/")
subprocess.call(f"aws s3 sync {url.as_uri()} {path}", shell=True)

Note depending on what your preferences you may want call the command in a different way (`subprocess.check_output()` or `shell=False`, but command is split by `' '`)

How do I download a folder from AWS S3 with CloudPathLib from a public bucket?

3 Answers3

Note depending on what your preferences you may want call the command in a different way (subprocess.check_output() or shell=False, but command is split by ' ')

Note depending on what your preferences you may want call the command in a different way (`subprocess.check_output()` or `shell=False`, but command is split by `' '`)