Which encoding is used for strings in Python 2.x?

Question

What is the default encoding used for encoding strings in python 2.x? I've read that there are two possible ways to declare a string.

string = 'this is a string'
unicode_string = u'this is a unicode string'

The second string is in Unicode. What is the encoding of the first string?

Answer 1

As per Python default/implicit string encodings and conversions (reciting its Py2 part concisely, to minimize duplication):

There are actually multiple independent "default" string encodings in Python 2, used by different parts of its functionality.

• Parsing the code and string literals:

  • str from a literal -- will contain raw bytes from the file, no transcoding is done
  • unicode from a literal -- the bytes from the file are decode'd with the file's "source encoding" which defaults to ascii
  • with unicode_literals future, all literals in the file are treated as Unicode literals

• Transcoding/type conversion:

  • str<->unicode type conversion and encode/decode w/o arguments are done with sys.getdefaultencoding()
    • which is ascii almost always, so any national characters will cause a UnicodeError
  • str can only be decode'd and unicode -- encode'd. Trying otherwise will involve an implicit type conversion (with the aforementioned result)

• I/O, including printing:

  • unicode -- encode'd with <file>.encoding if set, otherwise implicitly converted to str (with the aforementioned result)
  • str -- raw bytes are written to the stream, no transcoding is done. For national characters, a terminal will show different glyphs depending on its locale settings.

Answer 2

The literal answer is that they do not necessarily represent any particular encoding. In Python 2, a string is just an array of bytes, exactly like the bytes type in Python 3. For a string s you can call s.decode() to get a Unicode string, but you must* pass the encoding manually for exactly that reason. You could use a string to hold ASCII bytes, or characters from Windows code page 850 (which is a superset of ASCII), or UTF8 bytes, or even UTF16 bytes. The last case is interesting because even if the characters in that string are in the ASCII range, the bytes do not match the ASCII-encoded version (they will alternate with the null character). The string type is even suitable for bytes of some binary format that do not correspond to any encoded string e.g. the bytes of an image file.

A more practical answer is that often ASCII is assumed. For example, the literal string "xyz" will give a three byte string with the bytes corresponding to the ASCII encoding of those characters.

This ambiguity is the reason for the change in behaviours and conventions around strings in Python 3.

* As noted in CristiFati's answer, it is possible to omit the encoding= argument to decode, in which case ASCII will be assumed. My mistake.

Answer 3

As @ArthurTacca explained in his answer, a string ("this is a string") is just an array of bytes (0x74 0x68 0x69 0x73 0x20 0x69 0x73 0x20 0x61 0x20 0x73 0x74 0x72 0x69 0x6e 0x67), and its encoding makes no sense outside decoding context (when the bytes are interpreted).

Check out [Python 2.Docs]: sys.getdefaultencoding().

>>> sys.version
'2.7.10 (default, Mar  8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)]'
>>> sys.getdefaultencoding()
'ascii'
>>> "\xff".decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

Answer 4

The first string does not have an encoding. It is raw bytes. A convincing way to prove this to yourself is to change the encoding used to decode the source code to something else, using the coding declaration. This way you can visibly tell the difference between ASCII and bytes.

Save this to a .py file and execute it:

# coding: rot13

s0 =  "this is a string"
s1 = o"this is a string"
s2 = h"guvf vf n fgevat"

nffreg s0 == s1 == s2
cevag s0
cevag s1
cevag s2

This source is encoded in a simple letter substitution cipher. Letters in a-z A-Z are "rotated" by 13 places, other characters are unchanged. Since there are 26 letters in the alphabet, rotating twice is an identity transform. Note that the coding declaration itself is not rotated, see PEP 263 if you want to understand why.

• nffreg is an assert statement, saying that these three strings all compare equal.
• cevag is a print statement.
• s2 is a unicode string with rotated u prefix. The other two are bytestrings.

Now, let's change the handling of the first string, by introducing the unicode literals __future__ import. Note that this future statement itself must be rotated, or you'll get a syntax error. This alters the way the tokenizer/compiler combo will process the first object, as will become evident:

# coding: rot13
sebz __shgher__ vzcbeg havpbqr_yvgrenyf

s0 =  "guvf vf n fgevat"
s1 = o"this is a string"
s2 = h"guvf vf n fgevat"

nffreg s0 == s1 == s2
cevag s0
cevag s1
cevag s2

We needed to change the text from this is a string into guvf vf n fgevat in order for the assert statement to remain valid. This shows that the first string does not have an encoding.