Create CSV from irregular list of dictionaries in python

Question

I have a list of dictionaries like

[
  {'AB Code': 'Test AB Code', 'Created': '2020-08-04 13:20:55.196500+00:00'},
  {'AB Code': 'Test AB Code', 'Created': '2020-08-04 13:20:11.315136+00:00', 'Name': 'John Doe', 'Email': 'john@example.com', 'Phone No.': '1234567890', 'Age': '31'}
]

The two dictionary objects have irregular keys. I want to create a header for each new key and values beneath it.

The resultant CSV should be

AB Code, Created, Name, Email, Phone No., Age
Test AB Code, 2020-08-04 13:20:55.196500+00:00, '', '', '', ''
Test AB Code, 2020-08-04 13:20:55.196500+00:00, John Doe, john@example.com, 1234567890, 31

What I'm doing is

# header
d_ = []

# values
for index, item in enumerate(data):
  if index == 0:
    d_.append(list(item.keys()))
  d_.append(list(item.values()))

# Add to CSV
buffer = io.StringIO()
wr = csv.writer(buffer, quoting=csv.QUOTE_ALL)
wr.writerows(d_)

Which generates CSV

AB Code, Created
Test AB Code, 2020-08-04 13:20:55.196500+00:00, '', '', '', ''
Test AB Code, 2020-08-04 13:20:55.196500+00:00, John Doe, john@example.com, 1234567890, 31

`import pandas as pd; df = pd.DataFrame(lst);df.to_csv("data.csv", index=False)` — bigbounty, Aug 06 '20 at 08:58

score 2 · Accepted Answer · answered Aug 06 '20 at 09:26

There is answer using pandas provided by @bigbounty in comments to the question. Here is solution using just standard library

import csv
from collections import ChainMap

data = [
  {'AB Code': 'Test AB Code', 'Created': '2020-08-04 13:20:55.196500+00:00'},
  {'AB Code': 'Test AB Code', 'Created': '2020-08-04 13:20:11.315136+00:00', 'Name': 'John Doe', 'Email': 'john@example.com', 'Phone No.': '1234567890', 'Age': '31'}
]

keys = list(ChainMap(*data))
with open('spam.csv', 'w', newline='') as f:
    wrtr = csv.DictWriter(f, fieldnames=keys, quoting=csv.QUOTE_ALL)
    wrtr.writeheader()
    wrtr.writerows(data)

Also there is extended discussion on merging dicts, which may be somewhat relevant.

Create CSV from irregular list of dictionaries in python

1 Answers1