0

I am using the script below to download the entirety of an S3 bucket (using the answer from https://stackoverflow.com/users/9806031/konstantinos-katsantonis in Download a folder from S3 using Boto3).

Each object in the bucket is a csv file containing an identical structure: 4 fields. 1 timestamp, 2 strings, 1 float. Always in that order.

import boto3
import botocore
import os

s3 = boto3.resource("s3",
                  region_name='us-east-2',
                  aws_access_key_id = '',
                  aws_secret_access_key = ''
                  )
bucket_name = '',
s3_folder = '',
local_dir = r''

def download_s3_folder(bucket_name, s3_folder, local_dir):
    bucket = s3.Bucket(bucket_name)
    for obj in bucket.objects.filter(Prefix=s3_folder):
        target = obj.key if local_dir is None \
            else os.path.join(local_dir, os.path.relpath(obj.key, s3_folder))
        if not os.path.exists(os.path.dirname(target)):
            os.makedirs(os.path.dirname(target))
        if obj.key[-1] == '/':
            continue
        bucket.download_file(obj.key, target)
download_s3_folder(bucket_name, s3_folder, local_dir)

When I execute the script, I get the following error. I suspect this is a result of the presence of float.

TypeError: expected string or bytes-like object

What would be the best way to work around this?

satdaya
  • 59
  • 4
  • 1
    **Side-note:** For improved security, it is recommended never to put the Access Key and Secret Key into your code files. Instead, use the AWS CLI `aws configure` command to store the credentials in a configuration file. boto3 will automatically use the credentials from that file. – John Rotenstein Dec 16 '20 at 21:50
  • 1
    What "columns" are you referring to? Which line is generating the error? Please edit your question to tell us more about what you are doing and what problem you are experiencing so that we can try to reproduce the situation ourselves. – John Rotenstein Dec 16 '20 at 21:52
  • Thanks @JohnRotenstein. I made some changes that hopefully clarified some things. – satdaya Dec 17 '20 at 01:14

1 Answers1

0

These lines:

bucket_name = '',
s3_folder = '',

should be:

bucket_name = ''
s3_folder = ''

The comma at the end of the line was causing the string to become a tuple, which is not valid as a bucket name.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470