How are these strings represented internally in Python interpreter ? I don't understand

Question

# -*- coding: utf-8 -*-

a = 'éáűőúöüó€'
print type(a)    # <type 'str'>
print a          # éáűőúöüó€
print ord(a[-1]) # 172

Why is this working ? Shouldn't be this a SyntaxError: Non-ASCII character '\xc3' in file ... ? There are unicode literals in the string.

When I prefix it with u, the results are different:

# -*- coding: utf-8 -*-

a = u'éáűőúöüó€'
print type(a)    # <type 'unicode'>
print a          # éáűőúöüó€
print ord(a[-1]) # 8364

Why? What is the difference between the internal representations in python ? How can I see it myself ? :)

Why should it be a syntax error to have bytes in a byte string? — Wooble, Feb 12 '13 at 18:15
The first is a `str` object containing the UTF-8 bytes that are in the file. The second is a `unicode` object formed by decoding the UTF-8. Use `repr()` to see the difference. — Wooble, Feb 12 '13 at 18:19
Why the downvotes? This seems like a legitimate question. To ask any clearer would require knowledge of the answer. — Jon-Eric, Feb 12 '13 at 18:22

Martijn Pieters · Accepted Answer · 2013-02-12T18:25:26.527

There are unicode literals in the string

No, there are not. There are bytes in the string. Python simply goes with the bytes your editor saved to disk when you created the file.

When you prefixed the string with a u'', you signalled to python that you are creating a unicode object instead. Python now pays attention to the encoding you specified at the top of your source file, and it decodes the bytes in the source file to a unicode object based on the encoding you specified.

In both cases, your editor saved a series of bytes to a file, for the € character, the UTF-8 encoding is three bytes, represented in hexadecimal as E282AC. The last byte in the bytestring is thus AC, or 172 in decimal. Once you decode the last 3 bytes as UTF-8, they together become the Unicode codepoint U+20AC, which is 8364 in decimal.

You really should read up on Python and Unicode:

The Python Unicode HOWTO
Pragmatic Unicode by Ned Batchelder
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

Wow, great answer thanks ! I already read the first link still didn't understand the difference. Now it's crystal clear ! :) — kissgyorgy, Feb 12 '13 at 18:28

How are these strings represented internally in Python interpreter ? I don't understand

1 Answers1

Linked