Decode error with Pandas reading a .txt that only occurs on one compuer

Question

I have a comma-separated .txt file with French characters such as Vétérinaire and Désinfectant.

import pandas as pd
df = pd.read_csv('somefile.txt', sep=',', header=None, encoding='utf-8')

[Decode error - output not utf-8]

I have read many Q&A posts (including this) and tried many different encoding such as 'latin1' and 'utf-16', they didn't work. However, I tried to run the exact same script on the different Windows 10 computer with similar Python setup (all Python 3.6), it works perfectly fine in the other computer.

Edit: I tried this. Using encoding='cp1252' helps for some of the .txt files I want to import, but for a few .txt files, it gives the following error.

  File "C:\Program_Files_Extra\Anaconda3\lib\encodings\cp1252.py", line 15, in decode
return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 25: character maps to <undefined>

Edit: Trying to identify encoding from chardet

import chardet 
import pandas as pd
test_txt = 'somefile.txt'

rawdata = open(test_txt, 'rb').read()
result = chardet.detect(rawdata)
charenc = result['encoding']

print (charenc)

df = pd.read_csv(test_txt, sep=',', header=None, encoding=charenc)

print (df.head())

utf-8
[Decode error - output not utf-8]

You could try to guess the encoding of the files by using chardet, https://stackoverflow.com/questions/269060/is-there-a-python-library-function-which-attempts-to-guess-the-character-encodin — Dani Mesejo, Feb 14 '19 at 02:49
Very strange, I used chardet and it identified the problematic .txt as `utf-8`. — KubiK888, Feb 14 '19 at 02:54
Perhaps this could be useful: https://stackoverflow.com/a/27456542/4001592, but indeed is very strange. — Dani Mesejo, Feb 14 '19 at 03:00
Thanks but I saw that before, seems like it's not a solution for Python 3 - https://stackoverflow.com/questions/28127513/attributeerror-module-object-has-no-attribute-setdefaultencoding — KubiK888, Feb 14 '19 at 03:05

score 0 · Answer 1 · answered Feb 14 '19 at 03:24

Your program opens your files with a default encoding and that doesn't match the contents of the file you are trying to open.

Option 1: Decode the file contents to python string objects:

rawdata = open(test_txt, 'rb', encoding='UTF8').read()

Option 2: Open the csv file in an editor like Sublime Text and save it with utf-8 encoding to easily read the file through pandas.

Decode error with Pandas reading a .txt that only occurs on one compuer

1 Answers1