11

I have a dict that's feed with url response. Like:

>>> d
{
0: {'data': u'<p>found "\u62c9\u67cf \u591a\u516c \u56ed"</p>'}
1: {'data': u'<p>some other data</p>'}
...
}

While using xml.etree.ElementTree function on this data values (d[0]['data']) I get the most famous error message:

UnicodeEncodeError: 'ascii' codec can't encode characters...

What should I do to this Unicode string to make it suitable for ElementTree parser?

PS. Please don't send me links with Unicode & Python explanation. I read it all already unfortunately, and can't make use of it, as hopefully others can.

dda
  • 6,030
  • 2
  • 25
  • 34
theta
  • 24,593
  • 37
  • 119
  • 159

1 Answers1

25

You'll have to encode it manually, to UTF-8:

ElementTree.fromstring(d[0]['data'].encode('utf-8'))

as the API only takes encoded bytes as input. UTF-8 is a good default for such data.

It'll be able to decode to unicode again from there:

>>> from xml.etree import ElementTree
>>> p = ElementTree.fromstring(u'<p>found "\u62c9\u67cf \u591a\u516c \u56ed"</p>'.encode('utf8'))
>>> p.text
u'found "\u62c9\u67cf \u591a\u516c \u56ed"'
>>> print p.text
found "拉柏 多公 园"
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Yes, that was the first thing I tried and I always try. Problem is with `ElementTree.tostring`. Can you please try `ElementTree.tostring(p, method='text')` and tell why it doesn't work? Thanks – theta Nov 21 '12 at 12:54
  • 1
    Ah, sorry. It was too obvious. `.tostring()` has optional argument 'encoding' which is probably set to ascii by default, so adding `encoding='utf-8'` works. Cheers – theta Nov 21 '12 at 12:56
  • @theta: Hehe, just about to tell you about that. :-) – Martijn Pieters Nov 21 '12 at 12:57
  • for more information, check this issue: https://bugs.python.org/issue11033 – Roger Huang May 23 '18 at 05:15