Is Python 2.7 actually converting my string to UTF-8 or is the definition of isalnum() different across different machines?

Question

My sample.txt:

é Roméo et Juliette vécu heureux chaque après

My program:

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-

with open("test4", "r") as f:
        s = f.read()
        print(s)
        print(isinstance(s, unicode))
        print(s[0].isalnum())

My output:

é Roméo et Juliette vécu heureux chaque après

False
False

From Python isalpha() and scandics and How do I check if a string is unicode or ascii? lead me to believe that both statements should be true.

My hypotheses:

1. Emacs is using "iso-latin-1" as the file encoding, which is mucking things up

2. isalnum() depends on something other than encoding

3. Line 2 isn't working

My biggest worry is #2. I do not really care about the result of isalnum(), I just want the result to be consistent for different machines/people. Worst case, I can just roll my own isalnum(); but I am curious why I am experiencing this behaviour in the first place.

Also, I want to be sure my program understand UTF-8 encoded documents across different machines as well.

Any ideas of what is going on?

Answer 1

Strings (type str) in Python 2.7 are bytes. When you read text from a file, you get bytes, with possibly the line endings changed. Therefore, s is not an instance of type unicode.

On a str, tests like isalnum() assume that the string is ASCII text. ASCII is defined only for codes 0 to 127. Python has no idea, and can have no idea, what characters are represented by values outside this range, because the encoding is not known. é is not an ASCII character and therefore is not considered alphanumeric.

What you want to do is decode the byte string you've read to a Unicode string:

u = s.decode("utf8")

(assuming the string is written to the file in UTF8 encoding; if that doesn't work, you can try latin1 or cp437... the latter is what my terminal gives me on Windows 10)

When you do that, u[0].isalnum() is True and isinstance(u, unicode) is also True.

Python 3 works a little differently. You have to tell Python what encoding to use when you open the file. Then it translates the strings to Unicode from that encoding as you read them. All strings in Python 3 are Unicode; there's a separate type, bytes, for byte strings. You probably ought to use Python 3 for a lot of different reasons, but its more coherent handling of text is certainly one of those reasons.