39

I have a code such that:

a = "\u0432"
b = u"\u0432"
c = b"\u0432"
d = c.decode('utf8')

print(type(a), a)
print(type(b), b)
print(type(c), c)
print(type(d), d)

And output:

<class 'str'> в
<class 'str'> в
<class 'bytes'> b'\\u0432'
<class 'str'> \u0432

Why in the latter case I see a character code, instead of the character? How I can transform Byte string to Unicode string that in case of an output I saw the character, instead of its code?

Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251
Alex T
  • 489
  • 1
  • 6
  • 9

2 Answers2

63

In strings (or Unicode objects in Python 2), \u has a special meaning, namely saying, "here comes a Unicode character specified by it's Unicode ID". Hence u"\u0432" will result in the character в.

The b'' prefix tells you this is a sequence of 8-bit bytes, and bytes object has no Unicode characters, so the \u code has no special meaning. Hence, b"\u0432" is just the sequence of the bytes \,u,0,4,3 and 2.

Essentially you have an 8-bit string containing not a Unicode character, but the specification of a Unicode character.

You can convert this specification using the unicode escape encoder.

>>> c.decode('unicode_escape')
'в'
Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251
  • 1
    faced this problem when working with redis sets and trying to convert them to json. Redis returns a set of byte data. using `unicode_escape` worked perfectly – lukik Feb 15 '14 at 06:02
1

Loved Lennart's answer. It put me on the right track for solving the particular problem I had faced. What I added was the ability to produce html-compatible code for \u???? specifications in strings. Basically, only one line was needed:

results = results.replace('\\u','&#x')

This all came about from a need to convert JSON results to something that displays well in a browser. Here is some test code that is integrated with a cloud application:

# References:
# http://stackoverflow.com/questions/9746303/how-do-i-send-a-post-request-as-a-json
# https://docs.python.org/3/library/http.client.html
# http://docs.python-requests.org/en/v0.10.7/user/quickstart/#custom-headers
# http://stackoverflow.com/questions/606191/convert-bytes-to-a-python-string
# http://www.w3schools.com/charsets/ref_utf_punctuation.asp
# http://stackoverflow.com/questions/13837848/converting-byte-string-in-unicode-string

import urllib.request
import json

body = [ { "query": "co-development and language.name:English", "page": 1, "pageSize": 100 } ]
myurl = "https://core.ac.uk:443/api-v2/articles/search?metadata=true&fulltext=false&citations=false&similar=false&duplicate=false&urls=true&extractedUrls=false&faithfulMetadata=false&apiKey=SZYoqzk0Vx5QiEATgBPw1b842uypeXUv"
req = urllib.request.Request(myurl)
req.add_header('Content-Type', 'application/json; charset=utf-8')
jsondata = json.dumps(body)
jsondatabytes = jsondata.encode('utf-8') # needs to be bytes
req.add_header('Content-Length', len(jsondatabytes))
print ('\n', jsondatabytes, '\n')
response = urllib.request.urlopen(req, jsondatabytes)
results = response.read()
results = results.decode('utf-8')
results = results.replace('\\u','&#x') # produces html hex version of \u???? unicode characters
print(results)
SoothingMist
  • 29
  • 1