How can I get all whitespaces in UTF-8 in Python?

Question

Possible Duplicate:
In Python, how to list all characters matched by POSIX extended regex `[:space:]`?

How can I get a list of all whitespaces in UTF-8 in Python? Including non-breaking space etc. I'm using python 2.7.

Answer 1

unicodedata.category will tell you the category code for any given character; the characters you want have code Zs. There doesn't appear to be any way to extract a list of the characters within a category except by iterating over all of them:

>>> for c in xrange(sys.maxunicode+1):
...     u = unichr(c)
...     if unicodedata.category(u) == 'Zs':
...         sys.stdout.write("U+{:04X} {}\n".format(c, unicodedata.name(u)))
... 
U+0020 SPACE
U+00A0 NO-BREAK SPACE
U+1680 OGHAM SPACE MARK
U+180E MONGOLIAN VOWEL SEPARATOR
U+2000 EN QUAD
U+2001 EM QUAD
U+2002 EN SPACE
U+2003 EM SPACE
U+2004 THREE-PER-EM SPACE
U+2005 FOUR-PER-EM SPACE
U+2006 SIX-PER-EM SPACE
U+2007 FIGURE SPACE
U+2008 PUNCTUATION SPACE
U+2009 THIN SPACE
U+200A HAIR SPACE
U+202F NARROW NO-BREAK SPACE
U+205F MEDIUM MATHEMATICAL SPACE
U+3000 IDEOGRAPHIC SPACE

(Note: if you do this test using Python 3.4 or later, MONGOLIAN VOWEL SEPARATOR will not appear in the list. Python 2.7 shipped with data from Unicode 5.2; this character was reclassified as general category Cf ("formatting control") in Unicode 6.3, which is the version that Python 3.4 used for its data. See https://codeblog.jonskeet.uk/2014/12/01/when-is-an-identifier-not-an-identifier-attack-of-the-mongolian-vowel-separator/ and https://www.unicode.org/L2/L2013/13004-vowel-sep-change.pdf for more detail than you probably require.)

You may also want to include categories Zl and Zp, which adds

U+2028 LINE SEPARATOR
U+2029 PARAGRAPH SEPARATOR

And you almost certainly do want to include all of the ASCII control characters that are normally considered whitespace -- for historical reasons (I presume), these are in category Cc.

U+0009 CHARACTER TABULATION  ('\t')
U+000A LINE FEED (LF)        ('\n')
U+000B LINE TABULATION       ('\v')
U+000C FORM FEED (FF)        ('\r')
U+000D CARRIAGE RETURN (CR)  ('\f')

The other 60-odd Cc characters should not be considered whitespace, even if their official name makes it sound like they are whitespace. For instance, U+0085 NEXT LINE is almost never encountered in the wild with its official meaning; it's far more likely to be the result of an erroneous conversion from Windows-1252 to UTF-8 of U+2026 HORIZONTAL ELLIPSIS.

A closely-related question is "what does \s match in a Python regular expression?" Again the best available way to answer this question is to iterate over all characters:

>>> s = re.compile(ru"^\s$", re.UNICODE)
>>> for c in range(sys.maxunicode+1):
...   u = unichr(c)
...   if s.match(u):
...      sys.stdout.write("U+{:04X} {}\n".format(
...        c, unicodedata.name(u, "<name missing>")))
U+0009 <name missing>
U+000A <name missing>
U+000B <name missing>
U+000C <name missing>
U+000D <name missing>
U+001C <name missing>
U+001D <name missing>
U+001E <name missing>
U+001F <name missing>
U+0020 SPACE
U+0085 <name missing>
U+00A0 NO-BREAK SPACE
U+1680 OGHAM SPACE MARK
U+180E MONGOLIAN VOWEL SEPARATOR
U+2000 EN QUAD
U+2001 EM QUAD
U+2002 EN SPACE
U+2003 EM SPACE
U+2004 THREE-PER-EM SPACE
U+2005 FOUR-PER-EM SPACE
U+2006 SIX-PER-EM SPACE
U+2007 FIGURE SPACE
U+2008 PUNCTUATION SPACE
U+2009 THIN SPACE
U+200A HAIR SPACE
U+2028 LINE SEPARATOR
U+2029 PARAGRAPH SEPARATOR
U+202F NARROW NO-BREAK SPACE
U+205F MEDIUM MATHEMATICAL SPACE
U+3000 IDEOGRAPHIC SPACE

(I don't know why unicodedata.name doesn't know the control characters' names. Again, if you do this test using Python 3.4 or later, MONGOLIAN VOWEL SEPARATOR will not appear in the list.)

This is all of the Z* characters, all of the Cc characters that are generally agreed to be whitespace, and five extra characters that are not generally agreed to be whitespace, U+001C, U+001D, U+001E, U+001F, and U+0085. Inclusion of the last group is a bug, but a largely harmless one, since using those characters for anything is also a bug.

Answer 2

Zs category might not be enough:

#!/usr/bin/env python
import sys
import unicodedata

import regex # $ pip install regex

for i in xrange(sys.maxunicode + 1):
    u = unichr(i)
    if regex.match(u"[[:space:]]", u):
        try:
            name = unicodedata.name(u)
        except ValueError:
            name = ""
        print("{:9s} {} {}".format(repr(u), unicodedata.category(u), name))

Output

u'\t'     Cc 
u'\n'     Cc 
u'\x0b'   Cc 
u'\x0c'   Cc 
u'\r'     Cc 
u' '      Zs SPACE
u'\x85'   Cc 
u'\xa0'   Zs NO-BREAK SPACE
u'\u1680' Zs OGHAM SPACE MARK
u'\u180e' Zs MONGOLIAN VOWEL SEPARATOR
u'\u2000' Zs EN QUAD
u'\u2001' Zs EM QUAD
u'\u2002' Zs EN SPACE
u'\u2003' Zs EM SPACE
u'\u2004' Zs THREE-PER-EM SPACE
u'\u2005' Zs FOUR-PER-EM SPACE
u'\u2006' Zs SIX-PER-EM SPACE
u'\u2007' Zs FIGURE SPACE
u'\u2008' Zs PUNCTUATION SPACE
u'\u2009' Zs THIN SPACE
u'\u200a' Zs HAIR SPACE
u'\u2028' Zl LINE SEPARATOR
u'\u2029' Zp PARAGRAPH SEPARATOR
u'\u202f' Zs NARROW NO-BREAK SPACE
u'\u205f' Zs MEDIUM MATHEMATICAL SPACE
u'\u3000' Zs IDEOGRAPHIC SPACE

Answer 3

The list of offical space characters in the unicode database is defined through the 'Zs' categorie:

http://www.fileformat.info/info/unicode/category/Zs/list.htm

I am not sure if there is a functionality in Python's unicodedata module. I doubt: you can look over all characters and check their category against 'Zs'.