63

Python newbie getting a bit frustrated with the csv module. At this rate, it would have been easier if I wrote the file parser myself, but I want to do things the Pythonic way ....

I have written a little python script that should save my data into a CSV file.

Here is a snippet of my code:

import csv

wrtr = csv.writer(open('myfile.csv','wb'),delimiter=',', quotechar='"')

for row in rows:
    wrtr.writerow([row.field1,row.field2,row.field3])

The file myfile.csv is created successfully, yet it is empty - but has a lock on it, as its still being used by the Python process. It seems that the data has been written to the file in memory, but it has not yet been flushed to disk.

Since the Python process is holding a lock on the file, then I assume that I am responsible for releasing the lock. Here are my questions:

  1. How do I get python to flush to disk
  2. How do I close the file that was opened in the csv.writer() method?
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
skyeagle
  • 6,925
  • 16
  • 56
  • 71
  • 75
    """it would have been easier if I wrote the file parser myself""" -- I'll file that under "famous last words". – John Machin Oct 20 '10 at 11:08
  • The main intuition is that the file handle is still open after you write to the csv writer. If you remove the 'open' call outside and maintain a separate file handle which you pass into csv.writer(csvFile,...) then you can close that file handle when you're done and it would flush. The answer provided by Tim explains some good ways to do this. – mindthief Nov 15 '11 at 00:06
  • 1
    csvwriter is missing a flush method, this is an oversight. – Colonel Panic Apr 19 '15 at 21:04
  • I have seen there is a `file.synch()` method. I wonder how that is different from the `flush` command. – Charlie Parker Jan 30 '17 at 21:04

2 Answers2

109

Use

with open('myfile.csv','wb') as myfile:
    wrtr = csv.writer(myfile, delimiter=',', quotechar='"')
    for row in rows:
        wrtr.writerow([row.field1,row.field2,row.field3])
        myfile.flush() # whenever you want

or 

myfile = open('myfile.csv','wb')
wrtr = csv.writer(myfile, delimiter=',', quotechar='"')
for row in rows:
    wrtr.writerow([row.field1,row.field2,row.field3])
    myfile.flush() # whenever you want, and/or
myfile.close() # when you're done.

The nice thing about the first approach is that your file will also be automatically properly closed in case of an Exception.

If you want your file object to be anonymous, then it will only be closed when the program exits. When or whether it is flushed depends on the OS - so it might be never until exit.

Tim Pietzcker
  • 328,213
  • 58
  • 503
  • 561
  • 2
    I would not `flush()` in each iteration step when it's not necessary to monitor the file content during the itration `for row in rows`. – Dr. Jan-Philip Gehrcke May 25 '12 at 08:47
  • 4
    @Jan-PhilipGehrcke: That's why I wrote "whenever you want" in the comment after that command. – Tim Pietzcker May 25 '12 at 09:44
  • 1
    @Tim I understood that you meant that. Just wanted to make that clear for others :). – Dr. Jan-Philip Gehrcke May 25 '12 at 10:24
  • 1
    sorry if this is a newbie question, but does one have to explicitly flush for things to get written to disk? Or does the CSV writer does that automatically? I know there is this `file.sync()` function. Is that different from flushing? I'm a bit confused at this point. – Charlie Parker Jan 30 '17 at 21:03
  • @CharlieParker No, you don't need to explicitly flush what you are writing to make sure it's on disk. Usually your program will build up a buffer of information to write before actually doing it, in order to save computations. Flushing is basically saying "write your buffer now instead of later". – HitLuca Nov 26 '18 at 10:05
9

The flush() and close() methods of the file object. Or use with.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • 2
    But I am creating an 'anonymous' file object - so I have nothing to call flush and close against. Unless I am mistaken, this means that I will have to modify the code slightly to cleate a file object first before passing it to the csv.writer() function. I can then call flush() and close() on the file object as you suggested. If that is the case, then the documentation at http://docs.python.org/library/csv.html is indeed as useless/misleading as I think it is, and I will NEVER refer to it again. – skyeagle Oct 20 '10 at 10:18
  • Works fine here. It could be something screwy that Windows is doing. – Ignacio Vazquez-Abrams Oct 20 '10 at 10:22
  • 7
    "But I am creating an 'anonymous' file object". Don't. – S.Lott Oct 20 '10 at 10:50
  • 4
    To @skyeagle's credit, many of the examples in the docs he linked to do this. This is unpythonic, I'd say ("explicit is better than implicit"), but I wouldn't go so far as to take the consequence of never looking at the docs again. :) – Tim Pietzcker Oct 20 '10 at 11:04
  • @Tim: had to give you a +1 for seeing my point of view :). Regarding the never looking at the Python docs again, once the "red mist" has settled, I will be fine .... :) – skyeagle Oct 20 '10 at 11:12