trouble with unicodecsv reader in python

Question

I'm having trouble using the unicodecsv reader. I keep looking for different examples of how to use the module, but everyone keeps referencing the exact sample from the unicodecsv website (or some similar variation).

import unicodecsv as csv
from io import BytesIO
f = BytesIO()
w = csv.writer(f, encoding='utf-8')
_ = w.writerow((u'é', u'ñ'))
_ = f.seek(0)
r = csv.reader(f, encoding='utf-8')
next(r) == [u'é', u'ñ']
>>> True

For me this example makes too many assumptions about our understanding. It doesn't look like a csv file is being passed. I've completely missed the plot.

What I want to do is:

1. Read the first line of the csv file which are headers
2. Read the remaining lines and put them in a dictionary

My broken code:

import unicodecsv
#
i = 0
myCSV = "$_input.csv"
dic = {}
#
f = open(myCSV, "rb")
reader = unicodecsv.reader(f, delimiter=',')
strHeader = reader.next()
#
# read the first line of csv
# use custom function to parse the header
myHeader = FNC.PARSE_HEADER(strHeader)
#
# read the remaining lines
# put data into dictionary of class objects
for row in reader:
    i += 1
    dic[i] = cDATA(myHeader, row)

And, as expected, I get the 'UnicodeDecodeError'. Maybe the example above has the answers, but they are just completely going over my head.

Can someone please fix my code? I'm running out of hair to pull out.

I switched the reader line to:

reader = unicodecsv.reader(f, encoding='utf-8')

Traceback: for row in reader: File "C:\Python27\unicodecsv\py2.py", line 128 in next for value in row]

UnicodeDecodeError: 'utf8' codec can't decode byte 0x90 in position 48: invalide start byte

When I strictly print the data using:

f = open(myCSV, "rb")
reader = csv.reader(f, delimiter=',')
for row in reader:
    print(str[row[9]] + '\n')
    print(repr(row[9] + '\n')
>>> UTAS ? Offline
>>> 'UTAS ? Offline'

Answer 1

You need to declare the encoding of the input file when creating the reader, just like you did when creating the writer:

>>> import unicodecsv as csv
>>> with open('example.csv', 'wb') as f:
...     writer = csv.writer(f, encoding='utf-8')
...     writer.writerow(('heading0', 'heading1'))
...     writer.writerow((u'é', u'ñ'))
...     writer.writerow((u'ŋ', u'ŧ'))
... 
>>> with open('example.csv', 'rb') as f:
...     reader = csv.reader(f, encoding='utf-8')
...     headers = next(reader)
...     print headers
...     data = {i: v for (i, v) in enumerate(reader)}
...     print data
... 
[u'heading0', u'heading1']
{0: [u'\xe9', u'\xf1'], 1: [u'\u014b', u'\u0167']}

Printing the dictionary shows the escaped representation of the data, but you can see the characters by printing them individually:

>>> for v in data.values():
...     for s in v:
...         print s
... 
é
ñ
ŋ
ŧ

EDIT:

If the encoding of the file is unknown, then it's best to use some like chardet to determine the encoding before processing.

Answer 2

If your final goal is read csv file and convert data into dicts then I would recommend using csv.DictReader. DictRead will take care of reading header and converting rest of the rows into Dict (rowdicts). This uses CSV moduels, which contains lots of documentation/example available.

>>> import csv
>>> with open('names.csv') as csvfile:
...     reader = csv.DictReader(csvfile)
...     for row in reader:
...         print(row['first_name'], row['last_name'])

To get more clarity you check examples here https://docs.python.org/2/library/csv.html#csv.DictReader