Largest number of rows in a csv python can handle?

Question

import csv
with open("C:\Anaconda3\FalkParameters\AllModels.csv","r") as f:
    reader = csv.reader(f)
    listofModels = list(reader)

AllModels is a huge csv file (2.4 GB). I am not sure how many rows are in there, because I can't open it with any application (notepad, notepad++, excel, etc.).

The code above runs okay, but if I try to print "listofModels", the console starts printing, but at some point I get a "ValueError: I/O operation on closed file". Printing stops at different rows each time.

I was wondering if there is a way I can deal with huge csv files in python.

score 5 · Accepted Answer · answered Jan 26 '17 at 14:59

5

Don't convert it to list. Rather use generator instead. print one value/line at a time.

for line in reader:
    print line

Hope this helps !

answered Jan 26 '17 at 14:59

Jay Parikh

2,419
17
13

I'm sure it does! – Jean-François Fabre Jan 26 '17 at 15:00
Yes, because printing huge file at a time will be the reason for memory full so wont be able to print it. – Jay Parikh Jan 26 '17 at 15:02
Great. Thanks much! – DPdl Jan 26 '17 at 15:16

Largest number of rows in a csv python can handle?

1 Answers1

Linked