Why is DictWriter not Writing all rows in my Dictreader instance?

Question

I'm new to coding and by default new to Python, so please excuse my ignorance...I'm working on it.

I am trying to write some code (Python 2.7) to take specific headers from multiple CSV files and export them as a single file. Here is my code:

import csv, os

path = 'C:/Test/'

for fn in os.listdir(path):
    if ".csv" in fn:
        with open(fn, 'rb') as f:
            with open('C:/Test/fun/output.csv', 'wb') as fou:
                reader = csv.DictReader(f, delimiter=",", quotechar="|")
                writer = csv.DictWriter(fou, delimiter=",", quotechar="|", fieldnames=    ['sku', 'stock.qty', 'stock.is_in_stock'], extrasaction='ignore')
                headers = {} 
                for n in writer.fieldnames:
                    headers[n] = n
                writer.writerow(headers)
                for row in reader:
                    print row
                    writer.writerow(row)



    elif ".csv" not in fn:
        break

The print request for the reader instance seems to print all rows from multiple files. I am testing on 3 files with known rows. However, the DictWriter output file only has the rows from the last of the files read. It just doesn't make sense to me how I can print row and writerow and get different results. Obviously my DictWriter is incorrectly written but I do not see where. Probably obvious to most but I am puzzled.

You are opening the file to `w`rite (which overwrites it if it exists), not to `a`ppend (which adds to the end of it). — jonrsharpe, Apr 18 '14 at 12:15
Your method of writing headers is overly verbose; `writer.writeheaders()` would have taken care of that for you with one call. — Martijn Pieters, Apr 18 '14 at 12:16

Martijn Pieters · Accepted Answer · 2014-04-18T12:24:59.663

You are opening your target CSV file and clearing it for each matching CSV file you read. Opening the file in 'wb' mode clears the file each time.

Moreover, you break out of the loop as soon as you find a filename that is not a CSV file; you probably didn't want to do that at all; remove the else branch there.

Open the file just once, and continue to use it while looping over the directory, instead:

with open('C:/Test/fun/output.csv', 'wb') as fou:
    writer = csv.DictWriter(fou, delimiter=",", quotechar="|", fieldnames=    ['sku', 'stock.qty', 'stock.is_in_stock'], extrasaction='ignore')
    writer.writeheader()

    for fn in os.listdir(path):
        if ".csv" in fn:
            with open(fn, 'rb') as f:
                reader = csv.DictReader(f, delimiter=",", quotechar="|")
                for row in reader:
                    print row
                    writer.writerow(row)

I used the DictWriter.writeheader() method to write your fieldnames to the output file as an initial header.

Absolutely brilliant! Any logic that I was applying to the issue was too linear. read first then write...opening the Write instance first(once) does exactly what I wanted. Point taken about the writeheader...there is lots of misleading 'help' on the web. — dbell, Apr 18 '14 at 12:35
@dbell: Stack Overflow is helping to remedy the misleading 'help' problem, one vote at a time! :-) — Martijn Pieters, Apr 18 '14 at 12:36

Why is DictWriter not Writing all rows in my Dictreader instance?

1 Answers1

Linked