Problems with the encoding of several concatenated files in python?

Question

I have a folder of lots of .txt files in spanish and I decided to merge them in one .txt file as follows:

import os import shutil

def concatFiles():
    path = '/Users/user/Desktop/OpinionsTAG_txt/'
    files = os.listdir(path)
    with open("/Users/user/Desktop/concat_file.txt", "wb") as fo:
        for f in files:
            with open(os.path.join(path, f), "rb") as fi:
                shutil.copyfileobj(fi, fo)


if __name__ == "__main__":
    concatFiles()

The problem is that the output(i.e. concat_file) doesn't respect the character spanish encoding for example in the concat_file is dirección instead of dirección. Another thing is that I'm working in OS X, when i open the concat_file with sublime text it looks like this: 0000 0001 2000 0000 0000 0001 4000 0000 and when i open concat_file with text edit it looks as i wanted, why is this happenning and how can i solve it?.

Answer 1

Try using codecs, as suggested here: https://stackoverflow.com/a/19591815/4339369. This will allow you to read and write the files as UTF-8, which may solve your problem.

import codecs
import os
import shutil
def concatFiles():
    path = '/Users/user/Desktop/OpinionsTAG_txt/'
    files = os.listdir(path)
    with codecs.open("/Users/user/Desktop/concat_file.txt", "wb",encoding='utf8') as fo:
        for f in files:
            with codecs.open(os.path.join(path, f), "rb",encoding='utf8') as fi:
                shutil.copyfileobj(fi, fo)


if __name__ == "__main__":
    concatFiles()

A good overview of Unicode issues in Python 2.x can be found here: http://nedbatchelder.com/text/unipain.html.

Edit for anyone in the future: on OS X, you can usually determine a file's encoding by

file -I <filename>

from How do I determine file encoding in OSX?.