Does anyone know a simple function that converts existing csv files to UTF-8 encoding?

Question

I have huge csv files and they contain '\xc3\x84' style characters instead of German umlauts, because I scrapped HTML using BeautifulSoup and wrote it in the csv files using Python 2.7.8.

I managed to replace all those characters with the help of this: Python 2.7.1: How to Open, Edit and Close a CSV file

and now my code looks like this:

import csv

new_rows = []
umlaut = {'\\xc3\\x84': 'Ä', '\\xc3\\x96': 'Ö', '\\xc3\\x9c': 'Ü', '\\xc3\\xa4': 'ä', '\\xc3\\xb6': 'ö', '\\xc3\\xbc': 'ü'}

with open('file1.csv', 'r') as csvFile:
    reader = csv.reader(csvFile)
    for row in reader:
        new_row = row
        for key, value in umlaut.items():
            new_row = [ x.replace(key, value) for x in new_row ]
        new_rows.append(new_row)

with open('file2.csv', 'wb') as f:
    writer = csv.writer(f)
    writer.writerows(new_rows)

When I open the csv I see Köln instead of Köln and other "German umlaut" problems. I can solve this problem manually by opening the CSV file with notepad and then save it as UTF-8, but I want to do it automated with python.

I do not quite get how to use the UnicodeWriter:

https://docs.python.org/2/library/csv.html#examples

The answers and solutions I found here on stackoverflow are all a little bit complicated.

My question are, how would I use for example the UnicodeWriter right in my case? Do you know any super easy function that does something like file2.encode('utf-8')? If such an easy like function doesn' t exist in Python, then why doesn't it exists yet, because encoding errors are very common?

Answer 1

Instead of using your own mapping, you can use string-escape encoding:

>>> print '\\xc3\\x84'.decode('string-escape')
Ä

---

import csv

def iter_decode(it):
    for line in it:
        yield line.decode('string-escape')

with open('file1.csv') as csvFile, open('file2.csv', 'w') as f:
    reader = csv.reader(iter_decode(csvFile))
    writer = csv.writer(f)
    for row in reader:
        writer.writerow(row)

Answer 2

Given that you have a unicode writer from the docs :

class UnicodeWriter:
    """
    A CSV writer which will write rows to CSV file "f",
    which is encoded in the given encoding.
    """

    def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
        # Redirect output to a queue
        self.queue = cStringIO.StringIO()
        self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
        self.stream = f
        self.encoder = codecs.getincrementalencoder(encoding)()

    def writerow(self, row):
        self.writer.writerow([s.encode("utf-8") for s in row])
        # Fetch UTF-8 output from the queue ...
        data = self.queue.getvalue()
        data = data.decode("utf-8")
        # ... and reencode it into the target encoding
        data = self.encoder.encode(data)
        # write to the target stream
        self.stream.write(data)
        # empty queue
        self.queue.truncate(0)

    def writerows(self, rows):
        for row in rows:
            self.writerow(row)

use it like so:

from __future__ import unicode_lterals
import codecs
f = codecs.open("somefile.csv", mode='w', encoding='utf-8')
writer = UnicodeWriter(f)
for data in some_buffer:
    writer.writerow(data)