8

I have a string in python 3 that has several unicode representations in it, for example:

t = 'R\\u00f3is\\u00edn'

and I want to convert t so that it has the proper representation when I print it, ie:

>>> print(t)
Róisín

However I just get the original string back. I've tried re.sub and some others, but I can't seem to find a way that will change these characters without having to iterate over each one. What would be the easiest way to do so?

rptynan
  • 190
  • 2
  • 3
  • 8

3 Answers3

16

You want to use the built-in codec unicode_escape.

If t is already a bytes (an 8-bit string), it's as simple as this:

>>> print(t.decode('unicode_escape'))
Róisín

If t has already been decoded to Unicode, you can to encode it back to a bytes and then decode it this way. If you're sure that all of your Unicode characters have been escaped, it actually doesn't matter what codec you use to do the encode. Otherwise, you could try to get your original byte string back, but it's simpler, and probably safer, to just force any non-encoded characters to get encoded, and then they'll get decoded along with the already-encoded ones:

>>> print(t.encode('unicode_escape').decode('unicode_escape')
Róisín

In case you want to know how to do this kind of thing with regular expressions in the future, note that sub lets you pass a function instead of a pattern for the repl. And you can convert any hex string into an integer by calling int(hexstring, 16), and any integer into the corresponding Unicode character with chr (note that this is the one bit that's different in Python 2—you need unichr instead). So:

>>> re.sub(r'(\\u[0-9A-Fa-f]+)', lambda matchobj: chr(int(matchobj.group(0)[2:], 16)), t)
Róisín

Or, making it a bit more clear:

>>> def unescapematch(matchobj):
...     escapesequence = matchobj.group(0)
...     digits = escapesequence[2:]
...     ordinal = int(digits, 16)
...     char = chr(ordinal)
...     return char
>>> re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, t)
Róisín

The unicode_escape codec actually handles \U, \x, \X, octal (\066), and special-character (\n) sequences as well as just \u, and it implements the proper rules for reading only the appropriate max number of digits (4 for \u, 8 for \U, etc., so r'\\u22222' decodes to '∢2' rather than ''), and probably more things I haven't thought of. But this should give you the idea.

Mr. T
  • 11,960
  • 10
  • 32
  • 54
abarnert
  • 354,177
  • 51
  • 601
  • 671
  • If you used the previous version that encoded to UTF-8, don't; see the updated version, which should be safer when you have only-partly-escaped strings. – abarnert Dec 10 '12 at 02:15
  • I knew about re.sub passing functions, but I didn't know how to convert the string, this is perfect, thanks :) – rptynan Dec 10 '12 at 02:30
  • @rptynan: Did you not know about `chr`, or the second parameter to `int`? I'll update the answer so anyone who comes along later doesn't have any questions… – abarnert Dec 10 '12 at 02:33
  • I used to, but I'm re-learning python now and I had forgotten about it. – rptynan Dec 10 '12 at 02:50
  • @rptynan: Well, `chr` isn't something you use every day, so it's not surprising you didn't remember it. (I code in Python regularly, and I still often find myself wasting 5 minutes trying to remember what happened to `unichr` in 3.x before remembering that it was merged with `chr`…) – abarnert Dec 10 '12 at 03:21
0

First of all, it is rather confused what you what to convert to.

Just imagine that you may want to convert to 'o' and 'i'. In this case you can just make a map:

mp = {u'\u00f3':'o', u'\u00ed':'i'}

Than you may apply the replacement like:

t = u'R\u00f3is\u00edn'
for i in range(len(t)):
    if t[i] in mp: t[i]=mp[t[i]]
print t
Yunzhi Ma
  • 662
  • 6
  • 9
  • I'm pretty sure he wants to convert _any_ Unicode escape sequence `u'\\uXXXX'` to the Unicode character `u'\uXXXX'`. – abarnert Dec 10 '12 at 02:04
  • Yes, and preferable sub it back into the original string, but I can take care of that myself if neeeded. – rptynan Dec 10 '12 at 02:07
0

I apologize for posting as a second answer, I don't have the reputation to comment on abarnert's solution.

After using his function to process approximately 50K android strings I noticed that there is yet another small improvement possible for certain use-cases.

I changed the + to {1,4} to deal with the case where valid hex characters follow a 4-digit escape.

I also changed int(escapesequence) to read int(digits)

>>> def unescapematch(matchobj):
...     escapesequence = matchobj.group(0)
...     digits = escapesequence[2:]
...     ordinal = int(digits, 16)
...     char = unichr(ordinal)
...     return char

>>> print re.sub(r'(\\u[0-9A-Fa-f]{1,4})', unescapematch, "Wi\u2011Fi")
Wi‑Fi

>>> print re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, "Wi\u2011Fi")
Traceback (most recent call last):
  File "<pyshell#102>", line 1, in <module>
    print re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, "Wi\u2011Fi")
  File "C:\Python27\lib\re.py", line 151, in sub
     return _compile(pattern, flags).sub(repl, string, count)
  File "<pyshell#99>", line 5, in unescapematch
    char = unichr(ordinal)
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
Michael Baechle
  • 1