python csv encoding writerow

Question

Using web2py on windows 2008 server I have a following problem

I am creating csv document from json and when writting list to file i get the following error. It crashes on csv writerow

<type 'exceptions.UnicodeEncodeError'> 'ascii' codec can't encode character u'\\u010c'

It works ok on my computer. Windows 7 but on server I have an encoding problems

any suggestions? thank you

My code for creating file is the following

dataDict = json.loads(data.replace("'", "\""))
path = path
scriptName = os.path.join(path, id + 'script.txt')
file = open(scriptName, 'wb') 
output = csv.writer(file, delimiter='\t')

##Month hours
file.write("begin month_hours \r\n")
file.write("delavec    mesec    month_hours_min    month_hours_max\r\n")
for rec in dataDict["mandatory"]:
    output.writerow(rec)
file.write("\r\nend month_hours \r\n")

You are most likely using Python 3 locally, Python 2 on the server. — Martijn Pieters, Aug 18 '14 at 12:57
Any reason you're opening the file in 'wb' (write binary) instead of 'w' (write text)? — HSquirrel, Aug 18 '14 at 13:17
@Yebach: then on the server you are dealing with data that happens to be outside the ASCII range and you got lucky locally. — Martijn Pieters, Aug 18 '14 at 14:39
@HSquirrel: that's the correct handling in Python 2 for the CSV module, see that module's documentation. — Martijn Pieters, Aug 18 '14 at 14:40

Martijn Pieters · Answer 1 · 2018-05-04T08:38:41.667

6

JSON strings are always Unicode values and in Python 2 need to be encoded when writing to a CSV file. If you don't do this explicitly, Python will use the ASCII codec. That's fine if all your data contains text in the ASCII range but fails when you encounter data beyond that.

Pick a different encoding and encode explicitly; UTF-8 is a good encoding to pick:

for rec in dataDict["mandatory"]:
    output.writerow([unicode(c).encode('utf8') for c in rec])

I first convert all values to unicode(), in case you have data in there that's not already a unicode() value; numbers or booleans or None for example. The result is then explicitly encoded to UTF-8.

edited May 04 '18 at 08:38

answered Aug 18 '14 at 14:51

Martijn Pieters

1,048,767
296
4,058
3,343

Note, in Python3, replace `unicode()` with `str()` above – Wes May 03 '18 at 15:18
2

@Wes: In Python 3, you'd have opened the file in text mode and specified a codec, so this whole issue wouldn't even play. You would not need to encode data passed to the `csv.writer()` object **at all**. – Martijn Pieters May 04 '18 at 08:39

kdazzle · Answer 2 · 2021-12-03T19:06:25.233

It's been a while since this question was posted, so I'm guessing that the Python API has changed, but nowadays, you can actually open the file with an explicit encoding, which fixed a similar issue that I was having:

Changing this line in your code should do the trick. No need to make your code O(n2)ish with that inner loop.

open(scriptName, 'w', encoding='utf-8')

Or for the full example from the python docs:

import csv
with open('some.csv', newline='', encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)

python csv encoding writerow

2 Answers2