7

I want to download the whole bucket "my-bucket" from aws s3 storage, and save it in the path "./resources/my-bucket". So, I'm doing the following, importing boto3, creating a resource with the access_key_id and secret_access_key, then iterating over all the objects in the bucket and downloading them using the download_file api call:

import boto3
s3 = boto3.resource(
service_name="s3",
region_name="us-west-1",
aws_access_key_id="AWUHWIEDFB",
aws_secret_access_key="AOIERJGOERWHJO"
)

for o in s3.Bucket("my-bucket").objects.all():
    filename = o.key
    s3.Bucket("my-bucket").download_file(Key=filename, Filename="./resources/my-bucket/" + filename)

The s3 bucket my-bucket itself has a folder called "input" with only png and jpg files, and also has another .pt file in the root path.

But I'm getting the error:

FileNotFoundError: [Errno 2] No such file or directory: './resources/my-bucket/input/.F89bdcAc'

I don't know what .F89bdcAc means, and why boto3 is trying to find that file.

How to fix this?

zendevil
  • 899
  • 4
  • 13
  • 26
  • 2
    Please check [here](https://stackoverflow.com/questions/31918960/boto3-to-download-all-files-from-a-s3-bucket) – CK__ Sep 13 '20 at 16:57
  • 2
    You're going to need to create local subfolders as needed if any object has a key such as `folder1/folder2/` for the object with key `folder1/folder2/cat.png`. – jarmod Sep 13 '20 at 19:29
  • 2
    **Side-note:** There should never be a need to put access credentials in your code (it is bad for security). If the code is running on an Amazon EC2 instance, simply assign an IAM Role to the instance. If the code is running on your own computer, use the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/) `aws configure` command to store the credentials in a local configuration file. – John Rotenstein Sep 13 '20 at 21:09

2 Answers2

4

See the answer of this post : Boto3 to download all files from a S3 Bucket

It separates the processing of the directories in your bucket and the processing of the files (keys). Directories on your system are created on the fly with os.makedirs, you won't get this error anymore. And you ought to change your access key / secret !

Julien
  • 508
  • 2
  • 6
  • 13
0

It seems you don't have a directory structure that you are providing in the Filename param.

Try creating provided directory structure, i-e './resources/my-bucket/input/' and then the name of the file at the end, along with the extension for better.

Nomi
  • 185
  • 2
  • 13