Getting rid of unicode characters in a list

Question

I've tried str() and x.encode('UTF8'). Is there a quick and easy way to remove the unicode chars? My list looks like this:

mcd = [u'Chicken saut\xc3\xa9ed potatoes',  'Roasted lamb with mash potatoes', 'Rabbit casserole with tarragon, mushrooms and dijon mustard sauce. Served with mash potatoes']

The reason I am trying to get rid of the u's is becasue I'd like to copy this data onto a CSV file. It gives me an error like the one below when i try to do so...

UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-10: ordinal not in range(128)

I thought it would be easier just to remove unicode altogether.

Thanks in advance!

Answer 1

This works for me:

mcd = [u'Chicken saut\xc3\xa9ed potatoes',  'Roasted lamb with mash potatoes', 'Rabbit casserole with tarragon, mushrooms and dijon mustard sauce. Served with mash potatoes']

new = [str(m) for m in mcd]

for m,n in zip(mcd,new): # compare before and after
    print type(m), type(n)

OUT:

<type 'unicode'> <type 'str'>
<type 'str'> <type 'str'>
<type 'str'> <type 'str'>

If the above doesn't work (see convo in comments):

new = [m.encode('utf-8') for m in mcd]

Answer 2

The problem is probably that you are pressing enter instead of printing the result. This calls repr instead of str. Quoting the doc:

In the interactive interpreter, the output string is enclosed in quotes and special characters are escaped with backslashes. While this might sometimes look different from the input (the enclosing quotes could change), the two strings are equivalent. reference

Let me show you:

In [1]: mcd = [u'Chicken saut\xc3\xa9ed potatoes',  'Roasted lamb with mash potatoes', 'Rabbit casserole with tarragon, mushrooms and dijon mustard sauce. Served with mash potatoes']

In [2]: mcd[0]
Out[2]: u'Chicken saut\xc3\xa9ed potatoes'

In [3]: print repr(mcd[0])
u'Chicken saut\xc3\xa9ed potatoes'

In [4]: print mcd[0]  # Here will use my current OS encoding, i think utf8 in my case
Chicken sautéed potatoes

In [5]: print mcd[0].encode('utf8')  # yes! i was right
Chicken sautéed potatoes

You should choose the encoding type first, i think in this case you have to use latin1:

In [20]: print mcd[0].encode('latin1')
Chicken sautéed potatoes

Hope to have helped.

Edit: I hadn't seen the edit of the question, if you want to replace the characters, check this answer

Answer 3

If the strings you have obtained are the result of web site scraping, it appears that the site you got them off has an incorrect encoding setting.

It is fairly common for sites to specify charset=utf-8 and then have the site's content actually in some other character set (windows-1252 in particular) or vice versa. There is no simple, universal workaround for this phenomenon (also known as mojibake).

You might want to try with different scraping libraries -- most have some sort of tactic for identifying and coping with this scenario, but they have different success rates in different scenarios. If you are using BeautifulSoup, you might want to experiment with different parameters to the chardet back end.

Of course, if you only care about correctly scraping a single site, you can hard-code an override for the site's claimed character encoding.

Your question as such doesn't make much sense. It's not really clear what you are trying to accomplish. u'Chicken and sauted potatoes' is no more correct and only marginally less unappealing than u'Chicken and sautéed potatoes' (and in some ways more unappealing, because you can't tell that there was an attempt to make it right, although it wasn't competently executed).

If you get an encoding error because you are feeding Unicode to a file handle with an ASCII encoding, the correct solution for that is to specify an encoding other than ASCII (commonly, UTF-8) when opening the file for writing.