Python 3, pythonic way to get the string literal representation of bytes?

Question

There is similiar question, but the solution doesn't seem to work.

Say I've encoded a string:

>>> a = 'dada大大'.encode('utf-8')
>>> type(a)
<class 'bytes'>
>>> a
>>> b'dada\xe5\xa4\xa7\xe5\xa4\xa7'

What I want is something like this:

dada\xe5\xa4\xa7\xe5\xa4\xa7

str(a) doesn't work:

>>> str(a)
>>> "b'dada\\xe5\\xa4\\xa7\\xe5\\xa4\\xa7'"

I've tried redirect stdout to a variable, but still, I got "b'dada\\xe5\\xa4\\xa7\\xe5\\xa4\\xa7'".

I can deal with it using regular expression and get what I want, but I'm searching for a more pythonic way to do this. Any suggestions?

Answer 1

As you were so nice to mention your actual problem in a comment, I’ll update my answer once more to respond to that actually. The original answer can be seen below.

It's the string I post to Github Markdown API. This is the only way that unicode character can be accepted. I got the rendered html with the orignal character dada大大

The GitHub Markdown API requires you to send the data as JSON. JSON itself borrows the string escaping from JavaScript, which would be \u5927 for this character. When using the json module however, you don’t need to worry about that at all:

from urllib import urlopen
import json

text = 'dada大大'
data = json.dumps({ mode: 'markdown', 'text': text }).encode()
r = urlopen('https://api.github.com/markdown', data)

print(r.read().decode()) # <p>dada大大</p>

As you can see, the API accepts the encoded text without problems and correctly produces the correct output, without having to worry about the encoding.

Or when using the raw API with the requests library:

h = { 'Content-Type': 'text/plain' }
r = requests.post('https://api.github.com/markdown/raw', text.encode(), headers=h)

print(r.content.decode()) # <p>dada大大</p>

---

Original answer

>>> a = 'dada大大'.encode('utf-8')
>>> a
b'dada\xe5\xa4\xa7\xe5\xa4\xa7'
>>> str(a)
"b'dada\\xe5\\xa4\\xa7\\xe5\\xa4\\xa7'"
>>> str(a)[2:-1]
'dada\\xe5\\xa4\\xa7\\xe5\\xa4\\xa7'
>>> print(_)
dada\xe5\xa4\xa7\xe5\xa4\xa7

When you just do str(a) you will get the string representation of the bytes string. Of course, when you just use it like that in the interpreter, the interpreter will actually call repr on it to display it. And a string that contains backslashes will have them escaped as \\. That’s where those came from.

And finally, you have to strip of the b' and the trailing ' to get just the content of the string representation of the bytes string.

Side note: str() and repr() will produce the same result when used on bytes objects.

---

According to Poke's answer, what I need is preventing autoescaping of repr.

No, you don’t. There are no double backslashes in the final string. They only appear because when you enter stuff in your REPL, it will output the return values of things to the console after calling repr on them. But that does not mean, that the actual string suddenly got changed:

>>> s = str(a)[2:-1]
>>> len(s)
28
>>> list(s)
['d', 'a', 'd', 'a', '\\', 'x', 'e', '5', '\\', 'x', 'a', '4', '\\', 'x', 'a', '7', '\\', 'x', 'e', '5', '\\', 'x', 'a', '4', '\\', 'x', 'a', '7']

As you can see, there are not double backslashes in the string. Yes, you can see them again, but that’s again only because the return value of list(s) is being printed by the REPL. Each item of the list is a single character though, including the backslashes. They are just escaped again because '\' wouldn’t be a valid string.

>>> '\'
SyntaxError: EOL while scanning string literal
>>> '\\'
'\\'
>>> len('\\')
1

Answer 2

bytes is really an array of integers:

>>> a = 'dada大大'.encode() # 'utf-8' by default
>>> list(a)
[100, 97, 100, 97, 229, 164, 167, 229, 164, 167]

You can get the hex values of each of these using

>>> list(map(hex, a))
['0x64', '0x61', '0x64', '0x61', '0xe5', '0xa4', '0xa7', '0xe5', '0xa4', '0xa7']

And therefore

>>> list(chr(x) if x < 128 else hex(x) for x in a)
['d', 'a', 'd', 'a', '0xe5', '0xa4', '0xa7', '0xe5', '0xa4', '0xa7']

>>> print("".join(chr(x) if x < 128 else hex(x).replace("0", "\\") for x in a))
dada\xe5\xa4\xa7\xe5\xa4\xa7

Answer 3

OK, finally I found the solution, it's from Python Replace \\ with \

a = 'dada大大'.encode('utf-8')
b = str(a)[2:-1].encode('utf-8').decode('unicode_escape')

Maybe I should have explained what I want clearer.

Edit - My test result

>>> import requests
>>> text = 'dada大大'
>>> h = {'Content-Type': 'text/plain'}
>>> r = requests.post('https://api.github.com/markdown/raw', text.encode(), headers=h)
>>> print(r.content.decode())
{"message":"Invalid request media type (expecting 'text/plain')","documentation_url":"http://developer.github.com/v3/markdown/#render-a-markdown-document-in-raw-mode"}
>>> print(r.content.decode('utf-8'))
{"message":"Invalid request media type (expecting 'text/plain')","documentation_url":"http://developer.github.com/v3/markdown/#render-a-markdown-document-in-raw-mode"}
>>> r = requests.post('https://api.github.com/markdown/raw', text.encode('utf-8'), headers=h)
>>> print(r.content.decode('utf-8'))
{"message":"Invalid request media type (expecting 'text/plain')","documentation_url":"http://developer.github.com/v3/markdown/#render-a-markdown-document-in-raw-mode"}