1

I am trying to write multiple dictionaries to a csv file, where header (key) gets written only once and rows (values) are written based on the key. I was able to figure this out for two dictionaries, but what if I am getting multiple dictionaries that need to be written?

I am streaming tweets which get converted to json, so I am trying to end with a CSV file that is sorted by each JSON key. Here is a more detailed explanation of what I am trying to do (Writing multiple JSON to CSV in Python - Dictionary to CSV Here is what I am trying to end up with but with thousands of potential rows of data (preferably sorted by key if possible):

Table

Here is my basic code for two dictionaries:

import csv

my_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3', 'key4': 'value4', 'key5': 'value5', 'key6': 'value6', 'key7': 'value7'}
my_dict2 = {'key1': 'value1A', 'key2': 'value2A', 'key3': 'value3A', 'key4': 'value4A', 'key5': 'value5A', 'key6': 'value6A', 'key7': 'value7A'}

with open('mycsvfile.csv', 'wb') as f:
    w = csv.DictWriter(f, my_dict.keys())
    w.writeheader()
    w.writerow(my_dict)
    if my_dict.keys() == my_dict2.keys():
        w.writerow(my_dict2)

print my_dict

P.S. I am a begginer!

Community
  • 1
  • 1
verkter
  • 758
  • 4
  • 15
  • 29

1 Answers1

2

You can put the multiple dicts in a list. You then iterate over that list and only write the rows where the keys match. Something like this:

keys = ['key1', 'key2', 'key3', 'key4', 'key5'] # and so forth..

dictionaries = [my_dict, my_dict2] # Just an example, you probably build this list while you stream the tweets.

with open('mycsvfile.csv', 'wb') as f:
    w = csv.DictWriter(f, my_dict.keys())
    w.writeheader()
    for d in [x for x in dictionaries if x.keys() == keys]: # only matching keys.
       w.writerow(d)
msvalkon
  • 11,887
  • 2
  • 42
  • 38
  • This is great! But what if I have let's say hundreds of dictionaries that I am streaming in JSON format? How would you set up an input to the writer? I don't think that appending one giant list of dictionaries would work. – verkter Jun 13 '14 at 05:01
  • Can't you write the dictionaries as they come in then? You haven't shown *how* you get the data. – msvalkon Jun 13 '14 at 05:12
  • I am using tweepy to real-time stream the data from Twitter. Here is the code (http://stackoverflow.com/questions/24155319/writing-multiple-json-to-csv-in-python-dictionary-to-csv) – verkter Jun 13 '14 at 05:20
  • By the way, when I stream json (as in the method above) it only writes a header and 1st row. – verkter Jun 14 '14 at 04:47
  • @verkter open a new question regarding the streaming. It's impossible to answer without more details. – msvalkon Jun 16 '14 at 05:42