How do I specify a range of unicode characters

Question

How do I specify a range of unicode characters from ' ' (space) to \u00D7FF?

I have a regular expression like r'[\u0020-\u00D7FF]' and it won't compile saying that it's a bad range. I am new to Unicode regular expressions so I haven't had this problem before.

Is there a way to make this compile or a regular expression that I'm forgetting or haven't learned yet?

You're right - that does compile - I had assumed that was the portion of the regular expression I was working on that was having problems. I'll post the part that's causing me issues. — spig, Oct 01 '10 at 01:25

Josh Lee · Accepted Answer · 2012-09-10T11:29:17.777

The syntax of your unicode range will not do what you expect.

The raw r'' string prevents \u escapes from being parsed, and the regex engine will not do this. The only range in this set is [0-\]:

>>> re.compile(r'[\u0020-\u00d7ff]', re.DEBUG)
in
  literal 117
  literal 48
  literal 48
  literal 50
  range (48, 117)
  literal 48
  literal 48
  literal 100
  literal 55
  literal 102
  literal 102

Making it a Unicode literal causes \u parsing while leaving other backslashes alone (although that’s not a concern here), but the leading zeroes are messing it up. The syntax is \uxxxx or \Uxxxxxxxx, so it’s parsed as "\u00d7, f, f".
```
>>> re.compile(ur'[\u0020-\u00d7ff]', re.DEBUG)
in
  range (32, 215)
  literal 102
  literal 102
```

Removing the leading zeroes or switching to \U0000d7ff will fix it:

>>> re.compile(ur'[\u0020-\ud7ff]', re.DEBUG)
in
  range (32, 55295)

Thanks so much - I had no idea about all the unicode stuff. So, it has to be 4 digits with the lower case 'u' and 8 with the upper case 'U', right? — spig, Oct 01 '10 at 02:36
Note this answer is made in the context of Python 2. In Python 3, there is no difference between `r"example"` and `ur"example"`. — Flimm, Jan 22 '18 at 16:46

score 5 · Answer 2 · answered Oct 01 '10 at 01:33

5

If you're using Python 2.x, you should make sure you're specifying a unicode string (with u'', or the "unicode" built-in):

>>> r = re.compile(u'[\u0020-\uD7FF]')
>>> r.search(u'foo \uD7F0 bar')
<_sre.SRE_Match object at 0xb7084950>
r.search(u' ')
<_sre.SRE_Match object at 0xb7084b48>

Using raw strings (as you are, with r'') gives you the (ascii) string composed by "backstroke" + the letter "u" plus the number 0 plus...

answered Oct 01 '10 at 01:33

rbp

43,594
3
38
31

Do you need the '0' characters when the \u isn't quite long enough to be 4 long? Like for a space, you'd only have \u20 but you'd have to write \u0020?, right? – spig Oct 01 '10 at 02:39
I see you've already been answered, but yeah, that's right :) – rbp Oct 04 '10 at 12:33
2

What about **full Unicode**? For example, the Gothic block runs from U+10330 GOTHIC LETTER AHSA through U+1034A GOTHIC LETTER NINE HUNDRED but also includes the next five code points, currently unnamed? How do you in Python express what other programming languages allow you to use `\p{Gothic}`, `\p{Script=Gothic}`, `\p{InGothic}`, or `\p{Block=Gothic}` for? For example, in Perl, `/\pN/ && /\p{InGothic}/` yields U+10341 GOTHIC LETTER NINETY and U+1034A GOTHIC LETTER NINE HUNDRED. That doesn't work in Python, though; why not? – tchrist Nov 02 '10 at 12:36
1

If I understood correctly, you can use an uppercase U for larger codepoints, padding to 8 hexadecimal digits. For instance, in your example of U+10330 you'd use u'\U00010330' – rbp Nov 08 '10 at 21:38
2

@tchrist `That doesn't work in Python, though; why not?` The Python `re` module has never handled Unicode particularly well, although the third party regex library (https://pypi.org/project/regex/) works fine. – HackerBoss Nov 09 '18 at 05:13

How do I specify a range of unicode characters

2 Answers2

Linked

Related