Boto3 read a file content from S3 key line by line

Question

With boto3, you can read a file content from a location in S3, given a bucket name and the key, as per (this assumes a preliminary import boto3)

s3 = boto3.resource('s3')

content = s3.Object(BUCKET_NAME, S3_KEY).get()['Body'].read()

This returns a string type. The specific file I need to fetch happens to be a collection of dictionary-like objects, one per line. So it is not a JSON format. Instead of reading it as a string, I'd like to stream it as a file object and read it line by line; cannot find a way to do this other than downloading the file locally first as

s3 = boto3.resource('s3')

bucket = s3.Bucket(BUCKET_NAME)

filename = 'my-file'
bucket.download_file(S3_KEY, filename)

f = open('my-file')

What I'm asking is if it's possible to have this type of control on the file without having to download it locally first?

https://stackoverflow.com/a/16890018/6103001 - maybe that is enough. If not look for `botocore.response.StreamingBody` — Michał Zaborowski, Nov 29 '17 at 22:38

score 7 · Answer 1 · answered Feb 03 '20 at 16:30

I found .splitlines() worked for me...

txt_file = s3.Object(bucket, file).get()['Body'].read().decode('utf-8').splitlines()

Without the .splitlines() the whole blob of text was return and trying to iterate each line resulted in each char being iterated. With .splitlines() iteration by line was achievable.

In my example here I iterate through each line and compile it into a dict.

txt_file = s3.Object(bucket, file).get()['Body'].read().decode(
        'utf-8').splitlines()

for line in txt_file:
    arr = line.split()
    print(arr)

I don't know why this answer got that much votes. This approach is actually first downloads the whole file to local memory, then splits it to lines. The answer by @EnzoMolion is much better. — Alexander, Oct 23 '22 at 05:43

score 6 · Answer 2 · answered Aug 20 '21 at 15:20

You also can take advantage of StreamingBody's iter_lines method:

for line in s3.Object(bucket, file).get()['Body'].iter_lines():
    decoded_line = line.decode('utf-b') # if decoding is needed

That would consume less memory than reading the whole line at once and then split it

score 1 · Answer 3 · answered Sep 08 '19 at 11:19

The following comment from kooshiwoosh to a similar question provides a nice answer:

from io import TextIOWrapper
from gzip import GzipFile
...

# get StreamingBody from botocore.response
response = s3.get_object(Bucket=bucket, Key=key)
# if gzipped
gzipped = GzipFile(None, 'rb', fileobj=response['Body'])
data = TextIOWrapper(gzipped)

for line in data:
    # process line

score 0 · Answer 4 · answered Jan 03 '18 at 18:40

0

This will do the work:

bytes_to_read = 512

content = s3.Object(BUCKET_NAME, S3_KEY).get()['Body'].read(bytes_to_read)

answered Jan 03 '18 at 18:40

Hezi Halpert

9
1

10

This is only helpful if the OP knows how many bytes are in each line. The question gives no indication that this is the case. In fact, it would seem more likely that the line length is an unknown – Joey Harwood Jan 03 '18 at 19:01

score 0 · Answer 5 · answered Nov 04 '18 at 18:06

0

This works for me:

json_object = s3.get_object(Bucket = bucket, Key = json_file_name)
json_file_reader = json_object['Body'].read()
content = json.loads(json_file_reader)

answered Nov 04 '18 at 18:06

Harry

1,021
4
21
41

score 0 · Answer 6 · answered Mar 10 '20 at 10:22

As of now you have a possibility to use the download_fileobj function. Here an example for a CSV file:

import boto3
import csv

bucket           = 'my_bucket' 
file_key         = 'my_key/file.csv'
output_file_path = 'output.csv'

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)

#Dump binary in append mode
with open(output_file_path, 'ab') as file_object:
    bucket.download_fileobj(
        Key     = file_key, 
        Fileobj = file_object,
    )

#Read your file as usual
with open(output_file_path, 'r') as csvfile:
    lines = csv.reader(csvfile)
    for line in lines:
        doWhatEver(line[0])

Boto3 read a file content from S3 key line by line

6 Answers6