ElementTree and unicode

Question

I have this char in an xml file:

<data>
  <products>
      <color>fumè</color>
  </product>
</data>

I try to generate an instance of ElementTree with the following code:

string_data = open('file.xml')
x = ElementTree.fromstring(unicode(string_data.encode('utf-8')))

and I get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 185: ordinal not in range(128)

(NOTE: The position is not exact, I sampled the xml from a larger one).

How to solve it? Thanks

On a sidenote - your sample data is incorrect - *products* and *product* — Jon Clements, Sep 10 '12 at 10:40

gitaarik · Answer 1 · 2015-07-14T14:14:13.960

35

Might you have stumbled upon this problem while using Requests (HTTP for Humans), response.text decodes the response by default, you can use response.content to get the undecoded data, so ElementTree can decode it itself. Just remember to use the correct encoding.

More info: http://docs.python-requests.org/en/latest/user/quickstart/#response-content

edited Jul 14 '15 at 14:14

answered Dec 29 '13 at 13:00

gitaarik

42,736
12
98
105

1

In general, you should pass xml data (as bytes) directly to an XML parser unless `response.text` takes into account that `response.content` is XML and follows the corresponding standards e.g., reads the xml declaration fi any to find out the character encoding (it seems unlikely that `requests` would do that and it shouldn't). – jfs Jul 14 '15 at 18:37

score 15 · Answer 2 · edited Jun 20 '14 at 23:34

15

You need to decode utf-8 strings into a unicode object. So

string_data.encode('utf-8')

should be

string_data.decode('utf-8')

assuming string_data is actually an utf-8 string.

So to summarize: To get an utf-8 string from a unicode object you encode the unicode (using the utf-8 encoding), and to turn a string to a unicode object you decode the string using the respective encoding.

For more details on the concepts I suggest reading The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (not Python specific).

edited Jun 20 '14 at 23:34

Mike Akers

12,039
14
58
71

answered Sep 10 '12 at 10:30

Lukas Graf

30,317
8
77
92

this doesn't change anything, unfortunately – pistacchio Sep 10 '12 at 10:36
2

The OP's problem is that he/she is trying to handle decoding, instead of leaving it to ElementTree itself... – Martijn Pieters Sep 10 '12 at 10:37
1

@MartijnPieters: Absolutely, I wrote my answer while on the go, should've looked at the question a bit more carefully. While encoding a bytestring to get unicode is definitely wrong, it wasn't the (real) problem here. – Lukas Graf Sep 10 '12 at 18:30
You saved my life :). – Thang Nguyen Oct 28 '15 at 04:30

score 12 · Accepted Answer · edited Oct 09 '15 at 11:33

12

You do not need to decode XML for ElementTree to work. XML carries it's own encoding information (defaulting to UTF-8) and ElementTree does the work for you, outputting unicode:

>>> data = '''\
... <data>
...   <products>
...       <color>fumè</color>
...   </products>
... </data>
... '''
>>> x = ElementTree.fromstring(data)
>>> x[0][0].text
u'fum\xe8'

If your data is contained in a file(like) object, just pass the filename or file object directly to the ElementTree.parse() function:

x = ElementTree.parse('file.xml')

edited Oct 09 '15 at 11:33

msanders

5,739
1
29
30

answered Sep 10 '12 at 10:35

Martijn Pieters

1,048,767
296
4,058
3,343

2

Sadly there are times when we have XML that does not have embedded encoding information and Elementree is getting it wrong, returning strs with broken characters in. – Kylotan Mar 26 '15 at 15:17
3

@Kylotan: then those XML documents are at fault. The XML specification is *very clear* about this; the document is encoded as UTF8 unless specifically stated otherwise in the XML header. – Martijn Pieters Mar 26 '15 at 15:18
2

@Kylotan: you *can* override the XML declaration with an [`ElementTree.XMLParser()`](https://docs.python.org/2/library/xml.etree.elementtree.html#xml.etree.ElementTree.XMLParser) object passed in to the `ElementTree.parse()` function, use that for broken XML input. – Martijn Pieters Mar 26 '15 at 15:21
@Kylotan: that doesn't make my answer incorrect just because you have broken XML, however. – Martijn Pieters Mar 26 '15 at 15:21
Well, it's hard for me to know if there's anything wrong with the XML given that it seemed to render ok elsewhere but I have had problems with the output, that's all I know. (Would undo the downvote, but SO doesn't allow me to.) – Kylotan Mar 26 '15 at 15:30
@Kylotan: make sure you didn't make any errors with the Unicode output when extracting data from the XML file either; Unicode can be tricky and are not precise about encodings. – Martijn Pieters Mar 26 '15 at 15:33
@MartijnPieters FWIW I've just hit this in Python 2.6 on Windows. Have to pass a string into a function which then uses `fromstring`. Source is a UTF8 file read using `codecs.open()`. The solution does seem to be to force a conversion to utf8 in such situations. – Charlie Clark Mar 31 '15 at 08:57
@CharlieClark: `codecs.open()` produces a *Unicode* value, so ``, not a byte string. Yes, you'd have to encode back to UTF-8. – Martijn Pieters Mar 31 '15 at 08:59
I'm only experiencing problems with `xml.etree.ElementTree.fromstring()` in Python 2.6 on Windows. Other platforms (Mac, Linux) don't seem to need to conversion back to an encoded string. For future reference just in case anyone else comes across the same issue. – Charlie Clark Mar 31 '15 at 09:07

score 2 · Answer 4 · answered Sep 10 '12 at 10:34

2

Have you tried using the parse function, instead of opening the file... (which BTW would require a .read() after it for the .fromstring() to work...)

import xml.etree.ElementTree as ET

tree = ET.parse('file.xml')
root = tree.getroot()
# etc...

answered Sep 10 '12 at 10:34

Jon Clements

138,671
33
247
280

score 1 · Answer 5 · answered Sep 10 '12 at 10:28

1

The most likely your file is not UTF-8. è character can be from some other encoding, latin-1 for example.

answered Sep 10 '12 at 10:28

Maksym Polshcha

18,030
8
52
77

i made sure that the file is save with utf-8 encoding. – pistacchio Sep 10 '12 at 10:38
Tried the encoding 'cp-1250', that didn't work. 'latin-1' did. Thanks! – EvertW May 12 '20 at 08:20

score 1 · Answer 6 · edited Mar 10 '14 at 10:27

1

Function open() does not return a string. Instead use open('file.xml').read().

edited Mar 10 '14 at 10:27

HRM

2,097
6
23
37

answered Mar 10 '14 at 10:07

hejhula

11
1

ElementTree and unicode

6 Answers6

Linked

Related