Python won't write all data to csv file from several lists

Question

I am scraping an XML file and storing the data into several lists. Eventually I take the data from the lists and write it into a CSV file. The code runs without any errors, but when I check the data after, it seems that last the items from the lists weren't written to the file. Eventually there should be about 2000 rows in the file.

Here's my code:

with codecs.open("some_file.csv", 'w', encoding='utf-8') as file:
    writer = csv.writer(file, lineterminator='\n', delimiter=";")

    for a, b, c in zip(l1, l2, l3):
        writer.writerow([a, b, c])

I fould this topic but I already use with. What am I missing?

Why are you zipping a list with itself? a b and c will all be the same thing. — Carcigenicate, Nov 18 '16 at 19:33
Assuming that meant to say, `zip(l1, l2, l3)`, can you confirm that `l1`, `l2`, and `l3` are all the same length? If they are not the same length, you might want [`itertools.zip_longest()`](https://docs.python.org/3.0/library/itertools.html#itertools.zip_longest). — Robᵩ, Nov 18 '16 at 19:34
Are you terminating your program in any way? Try printing out values using `print` before your `writerow`. See if the problem is with the looping or with the `writer` — Patrick Haugh, Nov 18 '16 at 19:34
All the lists are the same length. When i dont´t use `zip` I get an error saying `too many values to unpack`. Debugging shows that all the data from xml are stored correctly in the lists, and nothing is missing. When I print out values before writing to file then the result is same as in the csv. Some values from the end are missing. So the problem could be with the looping. — RistoYlem, Nov 18 '16 at 19:52

score 0 · Answer 1 · answered Nov 18 '16 at 19:52

The code looks fine;

$ cat test.py
import codecs
import csv

l1 = ['hest', 'hat', 'test', 'more test']
l2 = ['hest2', 'hat', 'test', 'even more test']
l3 = ['hest3', 'hat', 'test', 'even even more test']

with codecs.open("some_file.csv", 'w', encoding='utf-8') as file:
    writer = csv.writer(file, lineterminator='\n', delimiter=";")

    for a, b, c in zip(l1, l2, l3):
    writer.writerow([a, b, c])

$ python test.py
$ cat some_file.csv 
hest;hest2;hest3
hat;hat;hat
test;test;test
more test;even more test;even even more test

lysdexia · Answer 2 · 2016-11-18T20:08:36.193

I think you might want to have a look at your three lists, and at zip.

If the lists are not of the same length, then zip will toss the offending ones out.

#!/usr/bin/env python

import csv
import codecs

L1 = ["foo", "bar", "bat", "baz"] # <-- this guy has one more! expect to be truncated
L2 = ["hoo", "hah", "hee"] # <--note, one field less!
L3 = range(2) #<--note, only two fields long! expect entire line to be dropped!

with codecs.open("test.csv", "w", encoding="utf-8") as f:
    writer = csv.writer(f, lineterminator="\n", delimiter=";")
    for row in zip(L1, L2, L3):
        writer.writerow(row)

Since you only have two values in L3, and the rest have three, zip will truncate your bad output.

~$ cat test.csv
foo;hoo;0
bar;hah;1

You will, of course, notice that I'm using zip to put out a single list value for each index in the three lists. This differs from your example. I don't know exactly what you are doing there, but if each of those "l" values represents a list, there is no real need to explicitly create those values, then add them to a list later.

Again, it would be nice to see an example of your inputs (if possible). One might have to insert None values (or empty strings, or whatever) when building those lists to get them to "match up".

L1 = ["foo", "bar", "bat", "baz"]
L2 = ["hoo", "hah", "hee", None]
L3 = range(2) + [None, None]

Which will probably give you what you want:

~$ cat test.csv
foo;hoo;0
bar;hah;1
bat;hee;
baz;;

Python won't write all data to csv file from several lists

2 Answers2