Python csvwriter only writes up to 132 rows

Question

I am trying to write data that is stored in a python dictionary to a csv file. For some reason the last 15 lines or so don't end up in the csv file. The python dictionary is formatted like this:

{name:{key1:value1,key2:value2},
name:{key1:value1,key2:value2},
name:{key1:value1,key2:value2},
name:{key1:value1,key2:value2,key3:value3},
}

I got this to work right before so I know its possible, I just don't remember what I did.

Here is my code:

featuresq =['name', 
'to_messages', 
'deferral_payments', 
'expenses', 
'poi', 
'deferred_income', 
'email_address', 
'long_term_incentive', 
'restricted_stock_deferred', 
'shared_receipt_with_poi', 
'loan_advances', 
'from_messages', 
'other', 
'director_fees', 
'bonus', 
'total_stock_value', 
'from_poi_to_this_person', 
'from_this_person_to_poi', 
'restricted_stock', 
'salary', 
'total_payments', 
'exercised_stock_options']

for name,line in data_dict.items():
    line["name"]=name
    row=[]
    for feature in featuresq:
    #   if feature in line.keys():\
        try:
            row.append(line[feature])
     #  else:
        except:
            row.append(float('NaN'))
    f.writerow(row)

It is very difficult to help you with the information you provide. Can you show us the value of futuresq and a link to what is being printed out in the csv file. — carlosdc, May 24 '15 at 03:36
Not a solution to your problem, but this will make your code easier to read - `row.append(line.get(feature, float('nan')))` , and you can remove the try-except block. — fixxxer, May 24 '15 at 03:52
What lines are you expecting but not being written to the file? — fixxxer, May 24 '15 at 03:58
well in this case it is lines 119 through 145. I'll post a link to the original data as a json file in a couple minutes. — Allan Visochek, May 24 '15 at 04:03
Are you closing the file before the script ends? http://stackoverflow.com/questions/15772275/writing-to-csv-from-list-write-row-seems-to-stop-in-a-strange-place — dting, May 24 '15 at 04:12
It seems your question has already been answered. http://stackoverflow.com/questions/8685809/python-writing-a-dictionary-to-a-csv-file-with-one-line-for-every-key-value — arshpreet, May 24 '15 at 04:19

fixxxer · Accepted Answer · 2015-05-24T05:19:35.220

0

Try using

    #This will close your file correctly if any untoward termination occurs. 
    with open('csv-file.csv','wb') as myfile: 
         # for testing purposes - you may flush each row to the file
         myfile.flush()

Secondly, I'd suggest that you use row.append(line.get(feature, float('nan'))) instead of the try and except. A dictionary has an in-built get function which would do exactly what you are doing with try-and-except and give you more crips code -

for name,line in data_dict.items():
    line["name"]=name
    row=[]
    for feature in featuresq:
        row.append(line.get(feature, float('nan')))
    f.writerow(row)
    FILE.flush()

edited May 24 '15 at 05:19

answered May 24 '15 at 04:17

fixxxer

15,568
15
58
76

yup, that works. Thanks for showing me the .get() method, super useful... – Allan Visochek May 24 '15 at 04:31
Flushing every loop is a bad idea. It'll slow down performance, and it's more complicated than the correct solution of just making sure to close files that you open. – abarnert May 24 '15 at 05:13
Also, the description is misleading. Flushing will _not_ close the file correctly if any untoward termination occurs. A `with` statement is how you close a file correctly even if there's an exception; `flush` leaves it still open, just with the writes from the last loop—and maybe part of the writes from the current loop—flushed. That's rarely what you want. – abarnert May 24 '15 at 05:15
Correct. Wondering how I missed that. – fixxxer May 24 '15 at 05:17

Python csvwriter only writes up to 132 rows

1 Answers1