Writing a list of dictionaries directly to S3 as csv

Question

I have a list of dictionaries like temp_dict = [{'id':'1', 'name':'john'},{'id':'2', 'name':'jake'},{'id':'3', 'name':'jacob'}] Is there a way using which I can directly write this dictionary as a csv file (pipe delimited) onto S3. I do not want to create a csv file on my local from this list and then copy that to S3.

What is the issue, exactly? Have you tried anything, done any research? Stack Overflow is not a free code writing service. See: [tour], [ask], [help/on-topic], https://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors. — AMC, Feb 13 '20 at 20:53

score 5 · Answer 1 · edited Oct 06 '20 at 12:57

Here is a solution which is a little bit more elegant and very fast particularly with large lists of dictionaries.

import io
import csv

data = [{"param1": 1, "param2": 2}, {"param1": 1, "param2": 3}]

stream = io.StringIO()
headers = list(data[0].keys())
writer = csv.DictWriter(stream, fieldnames=headers)
writer.writeheader()
writer.writerows(data)

csv_string_object = stream.getvalue()

Using this string object which is a representation of your CSV file content, you can directly insert it into S3 in whichever manner you prefer via boto3.

session = boto3.session.Session(profile_name=<your_profile_name>)
resource = session.resource("s3")
resource.Object(<s3_bucket>, <s3_key>).put(Body=csv_string_object)

Beware, your CSV file is now loaded in memory and might crash with large quantities of data.

Care to elaborate? I am pretty sure it works very well. – gabzo Jul 30 '21 at 14:08 — gabzo, Jul 30 '21 at 14:08

score 1 · Accepted Answer · answered Feb 13 '20 at 20:45

I think this would be helpful:

import csv

class Pipe:
    value = ""
    def write(self, text):
        self.value = self.value + text

temp_dict = [{'id':'1', 'name':'john'},{'id':'2', 'name':'jake'},{'id':'3', 'name':'jacob'}]

pipe = Pipe()
writer = csv.DictWriter(pipe, temp_dict[0].keys())
for entry in temp_dict:
    writer.writerow(entry)

print(pipe.value)

1,john
2,jake
3,jacob

Basically, we write a class to imitate a file object in write mode, then we create an instance, and pass it to the DictWriter. At the end, we get the csv text from pipe.value.

score 0 · Answer 3 · answered Feb 13 '20 at 20:38

0

This could solve your issue.

Convert your in-memory object in to binary stream and pass onto s3 using boto3.

How to write a file or data to an S3 object using boto3

answered Feb 13 '20 at 20:38

gsb22

2,112
2
10
25

Hi Gauri, can you please share some sample code. I am not sure how to convert list of dictionaries into a csv stream – Akshay Jagadale Feb 13 '20 at 20:47

Writing a list of dictionaries directly to S3 as csv

3 Answers3