1

I need to upload URLs to an s3 bucket and am using boto3. I thought I had a solution with this question: How to save S3 object to a file using boto3 but when I go to download the files, I'm still getting errors. The goal is for them to download as audio files, not URLs. My code:

    for row in list_reader:
        media_id = row['mediaId']
        external_id = row['externalId']
        with open('10-17_res1.csv', 'a') as results_file:
            file_is_empty = os.stat('10-17_res1.csv').st_size == 0
            results_writer = csv.writer(
            results_file, delimiter = ',', quotechar = '"'
            )
            if file_is_empty:
                results_writer.writerow(['fileURL','key', 'mediaId','externalId'])

            key = 'corpora/' + external_id + '/' + external_id + '.flac'
            bucketname = 'my_bucket'

            media_stream = media.get_item(media_id)
            stream_url = media_stream['streams'][0]['streamLocation']


            fake_handle = StringIO(stream_url)
            s3c.put_object(Bucket=bucketname, Key=key, Body=fake_handle.read())

My question is, what do I need to change so that the file is saved in s3 as an audio file, not a URL?

topplethepat
  • 531
  • 6
  • 23

1 Answers1

1

I solved this by using the smart_open module:

        with smart_open.open(stream_url, 'rb',buffering=0) as f:
            s3.put_object(Bucket=bucketname, Key=key, Body=f.read())

Note that it won't work without the 'buffering=0' parameter.

topplethepat
  • 531
  • 6
  • 23