Does python 3.1.3 support unicode in csv module?

Question

I have been using python 2.6. While I was writing a python program to process the query result ( in csv format ) from sql server. I found it does not support unicode.

When I run the program with csv file, a error poped up saying:

    for row in csvReader:
Error: line contains NULL byte

After I save the csv file in ANSI/ASCII format with Ultraedit, the program is running okay.

I tried to include the encoding option, but it failed:

csvReader = csv.reader(open(fname, mode='rb', encoding='unicode'), delimiter=',')
TypeError: 'encoding' is an invalid keyword argument for this function

csvReader = csv.reader(open(fname, mode='rb', encoding='utf-8'), delimiter=',')
TypeError: 'encoding' is an invalid keyword argument for this function

I wonder if python 3 support this unicode reading. It can save me a lot of work.

Python 3.1.3's `open` *definitely* supports and `encoding=` argument, so either you're using a different version of Python, or you have accidentally overwritten the `open` function. — David Wolever, Feb 29 '12 at 07:02
I said I am using python 2.6. So Python 3.1.3 won't have this problem? — lamwaiman1988, Feb 29 '12 at 07:05
Python 3 still have this problem. Apparently there are a NUL byte there messing up python, no matter it is python 3 or python 2. — lamwaiman1988, Feb 29 '12 at 07:26
Oh, sorry, it wasn't clear that you were using Python 2.6. See my updated answer. — David Wolever, Feb 29 '12 at 19:00

David Wolever · Answer 1 · 2012-02-29T19:04:10.417

7

Python 3 definitely supports unicode. My guess is that you specified the wrong (or no?) encoding when you opened the CSV file for reading. See: http://docs.python.org/release/3.1.3/library/functions.html#open

And try something like:

reader = csv.reader(open("foo.csv", encoding="utf-8"))

Edit: If you are using Python 2.6, you can achieve the same result with:

import codecs
reader = csv.reader(codecs.open("foo.csv", encoding="utf-8"))

HOWEVER if you're getting null bytes, your file may be encoded using "utf-16", so try that if the file can't be decoded using utf-8.

edited Feb 29 '12 at 19:04

answered Feb 29 '12 at 06:55

David Wolever

148,955
89
346
502

I tried to specified the encoding, but it return an error. Please check my edit. – lamwaiman1988 Feb 29 '12 at 06:58

score 2 · Answer 2 · edited May 23 '17 at 12:08

2

A similar question is already answered Python CSV error: line contains NULL byte

Also, Try to open it in 'rb' mode instead of 'rU'

edited May 23 '17 at 12:08

Community

1
1

answered Feb 29 '12 at 06:57

jerrymouse

16,964
16
76
97

Does python 3.1.3 support unicode in csv module?

2 Answers2

Linked