3

I'm trying to import a csv file using:

data = pd.read_csv("filename.csv")

I get the following error: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 2: invalid start byte".

The answer in this question: UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c might work, but I am not sure how to implement it (I can't comment on the answer because I don't have enough reputation yet).

Any help would be appreciated.

Edit: The issue seems to be linked to the fact that I have a degree symbol. It would be fine for me if during import this issue is just skipped.

6135105654
  • 202
  • 2
  • 5
  • 13
  • 1
    You'll have to post a link to your raw data – EdChum Jan 10 '19 at 16:57
  • Thanks for the reply @EdChum, managed to figure out a work around which was to skip the row which had the offending symbol: data = pd.read_csv("filename.csv", skiprows=[1]) – 6135105654 Jan 10 '19 at 17:12

3 Answers3

16

If you face an encoding error due to encoding on your file not being the default as mentioned by the pd.read_csv() docs , you can find the encoding of the file by first installing chardet followed by the below code:

import chardet    
rawdata = open('D:\\path\\file.csv', 'rb').read()
result = chardet.detect(rawdata)
charenc = result['encoding']
print(charenc)

This will give you the encoding of the file.

Once you have the encoding, you can read as :

pd.read_csv('D:\\path\\file.csv',encoding = 'encoding you found')

or

pd.read_csv(r'D:\path\file.csv',encoding = 'encoding you found')

You will get the list of all encoding here

Hope you find this useful.

Ritz
  • 238
  • 1
  • 6
  • 4
    Thanks for the response. Really enjoying this solution, it has worked perfectly, and by placing charenc into the read statement it's done automatically: pd.read_csv('D:\\path\\file.csv',encoding = charenc) – 6135105654 Jan 10 '19 at 21:42
  • Great solution, much better than hard coding a encoding, while often you don't know this in advance! – moojen Feb 22 '22 at 16:06
0

I solved mine by simply going back to the excel sheet and saving using the format 'CSV UTF-8'

Elias
  • 1
-1

You can use the encoding argument of the pandas function read_csv.

It can look something like that if it really need to be encode in utf-8.

import pandas as pd
df = pd.read_csv("filename.csv", encoding = 'utf_8')
Charles_de_Montigny
  • 69
  • 6