Error loading large csv file "'utf-8' codec can not decrypt" in python

Question

I have 1 GB csv file and I can not read that log file and give same error in both python and pandas code in my csv file, it is not a value of more than one column because there is only a single column value and All of my CSV values is number

with open("/Users/kiya/sep_sent.csv", encoding='utf-8') as f:
for i in f: 
   print(i.strip())

another method:

with open("/Users/kiya/sep_sent.csv",encoding='cp1252') as f:
    for i in f:
      print(i.strip())

Traceback (most recent call last):
  File "/Users/kiya/test8.py", line 5, in <module>
    for i in f:
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 159: character maps to <undefined>

pandas code:

import pandas as pd
df = pd.read_csv("/Users/kiya/sep_sent.csv", encoding="utf-8")
print(df)

my csv value like:

error:

0
0
0
0
0
Traceback (most recent call last):
  File "/Users/kiya//test8.py", line 4, in <module>
    for i in f:
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 52: invalid start byte

Try with somthing smaller .. see my comment below, your terminal is `cp1252` and in the csv file are utf-8 characters which are not printable in that terminal, now you got this exception only when printing. — return42, Oct 03 '18 at 13:48
Test your terminal encoding with `python -c "import sys; print(sys.stdout.encoding)"` — return42, Oct 03 '18 at 13:50
You need to figure out which encoding was used to encode the file. Or try [other encodings](https://docs.python.org/3/library/codecs.html#standard-encodings) at random. `'latin_1'` or the `'iso8859_XX'` variants might work but it woud mean there are control characters in your file. — Stop harming Monica, Oct 03 '18 at 13:58
without the csv-file its hard to help, I switch of now .. good luck. — return42, Oct 03 '18 at 14:03
Take a loke here https://stackoverflow.com/a/46000253/300130, as @goyo mentioned, the data is not encoded utf-8. — return42, Oct 03 '18 at 14:19

score 0 · Answer 1 · answered Oct 03 '18 at 13:16

0

You can pass the encoding argument to read_csv as well

df = pd.read_csv("/Users/kiya/sep_sent.csv", encoding="utf-8")

answered Oct 03 '18 at 13:16

Arjun Venkatraman

239
2
10

It shows that same error – warezers Oct 03 '18 at 13:30
Looks like you have non unicode characters in your text. See @Goyo's answer in comments above. To find out where the issue is in the file, you might want to consider chunking up the file into smaller units to identify which unit the problem is in – Arjun Venkatraman Oct 13 '18 at 05:32

score -1 · Answer 2 · answered Oct 03 '18 at 13:09

-1

Open the file with utf-8 encoding and it should work:

with open("/Users/kiya/sep_sent.csv", encoding='utf-8') as f:
    for i in f: 
       print(i.strip())

answered Oct 03 '18 at 13:09

return42

543
3
10

Actually the error message is saying that `utf-8` doesn't work. – Stop harming Monica Oct 03 '18 at 13:20
It shows that same error – warezers Oct 03 '18 at 13:30
OK, but now the you get output `0` before it was `b0` which was binray zero, now you got string character `0` .. compare https://stackoverflow.com/posts/52627668/revisions – return42 Oct 03 '18 at 13:38
ya but It is not running completely and an error is displayed – warezers Oct 03 '18 at 13:41
It seems your terminal is using `cp1252` (see traceback) and in the csv file there are chracters which are not printable on such a codepage. – return42 Oct 03 '18 at 13:44
Is it possible to skip that values? Since the CSV file is large in capacity and can not be opened in Excel – warezers Oct 03 '18 at 13:51
@return42 The traceback says that the error is triggered by `for i in f:` so when reading the file not when printing. – Stop harming Monica Oct 03 '18 at 13:52
@goyo: there is a print at this point and this print statement is (now) causing the excepition .. typical encoding mistakes – return42 Oct 03 '18 at 13:55
@return42 I don't understand what you mean, I cannot see `print` in the tracebacks. They clearly say that `for i in f:` is the user code that triggers the error. – Stop harming Monica Oct 03 '18 at 14:02
@goyo: sorry, my mistake .. the example was edited so often, I lost control ;) – return42 Oct 03 '18 at 14:10

Error loading large csv file "'utf-8' codec can not decrypt" in python

2 Answers2