Decoding UTF8 literals in a CSV file

Question

Question:

Does anyone know how I could transform this b"it\\xe2\\x80\\x99s time to eat" into this it's time to eat

More details & my code:

Hello everyone,

I'm currently working with a CSV file which full of rows with UTF8 literals in them, for example:

b"it\xe2\x80\x99s time to eat"

The end goal is to to get something like this:

it's time to eat

To achieve this I have tried using the following code:

import pandas as pd


file_open = pd.read_csv("/Users/Downloads/tweets.csv")

file_open["text"]=file_open["text"].str.replace("b\'", "")

file_open["text"]=file_open["text"].str.encode('ascii').astype(str)

file_open["text"]=file_open["text"].str.replace("b\"", "")[:-1]

print(file_open["text"])

After running the code the row that I took as an example is printed out as:

it\xe2\x80\x99s time to eat

I have tried solving this issue using the following code to open the CSV file:

file_open = pd.read_csv("/Users/Downloads/tweets.csv", encoding = "utf-8")

which printed out the example row in the following manner:

it\xe2\x80\x99s time to eat

and I have also tried decoding the rows using this:

file_open["text"]=file_open["text"].str.decode('utf-8')

Which gave me the following error:

AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas

Thank you very much in advance for your help.

Can you elaborate on why `pd.read_csv(..., encoding="utf8")` failed? — jedwards, Jun 24 '18 at 16:47
"all of these attempts failed" is not particularly helpful. Please provide the details of *how* they failed, per the guidelines for a [MCVE]. — ShadowRanger, Jun 24 '18 at 16:48
@ShadowRanger My bad, I just added the part on how the failed and the output I got. — Nazim Kerimbekov, Jun 24 '18 at 16:51
Please show how the CSV file is created in the first place. That process is buggy and the problem should be fixed there instead. See also https://stackoverflow.com/questions/51885694/how-to-decode-backslash-scapes-strings-in-python — Karl Knechtel, Aug 05 '22 at 02:26

jedwards · Accepted Answer · 2018-06-24T17:28:11.983

b"it\\xe2\\x80\\x99s time to eat" sounds like your file contains an escaped encoding.

In general, you can convert this to a proper Python3 string with something like:

x = b"it\\xe2\\x80\\x99s time to eat"
x = x.decode('unicode-escape').encode('latin1').decode('utf8')
print(x)     # it’s time to eat

(Use of .encode('latin1') explained here)

So, if after you use pd.read_csv(..., encoding="utf8") you still have escaped strings, you can do something like:

pd.read_csv(..., encoding="unicode-escape")
# ...
# Now, your values will be strings but improperly decoded:
#    itâs time to eat
#
# So we encode to bytes then decode properly:
val = val.encode('latin1').decode('utf8')
print(val)   # it’s time to eat

But I think it's probably better to do this to the whole file instead of to each value individually, for example with StringIO (if the file isn't too big):

from io import StringIO

# Read the csv file into a StringIO object
sio = StringIO()
with open('yourfile.csv', 'r', encoding='unicode-escape') as f:
    for line in f:
        line = line.encode('latin1').decode('utf8')
        sio.write(line)
sio.seek(0)    # Reset file pointer to the beginning

# Call read_csv, passing the StringIO object
df = pd.read_csv(sio, encoding="utf8")

Thank you very much for your answer, this completely solves my problem, what I don't really understand is what you mean by "escaped encoding". — Nazim Kerimbekov, Jun 24 '18 at 17:43
The double backslashes indicate that somewhere before the csv was written to file, the backslashes were escaped (doubled), and *then* the file was written like this. — jedwards, Jun 24 '18 at 17:47
It would be better to fix the code that created the file in the first place. — Karl Knechtel, Aug 05 '22 at 02:26

Decoding UTF8 literals in a CSV file

1 Answers1