trying to change delimiters in large csv files resulting in memory errors (python 3)

Asked Jun 08 '19 at 14:38

Active Jun 08 '19 at 14:38

Viewed 284 times

I have a bunch of comma-delimited files that I am trying to change to pipe-delimited files.

I am following the example provided here: Python CSV change separator

Here is my code:

print("setting new delimiter...")
reader = list(csv.reader(open(localfile, "rU"), delimiter=','))
writer = csv.writer(open(localfile, 'w'), delimiter='|', lineterminator='\n')
writer.writerows(row for row in reader)

I can't tell if memory usage is cumulative, or because of a specific filesize, but either way, on my third file, I get a memory error.

since the third file is nearly the same size as the first two, it appears cumulative.

Is there a better way in doing this? Thanks

asked Jun 08 '19 at 14:38

arcee123

2

Dont use `list` on the `csv.reader`... it's iterable... then just use `writer.writerows(reader)`... (you don't need a gen-exp there) – Jon Clements Jun 08 '19 at 14:42
You never actually close the file btw – roganjosh Jun 08 '19 at 14:48
should I? just `reader.close()` – arcee123 Jun 08 '19 at 23:11

trying to change delimiters in large csv files resulting in memory errors (python 3)

0 Answers0