0
a1 = b'\x01\x02\x41'

\x41 is A

and when I print str(a1), I get

b'\x01\x02A'

how could I get the original hex string?

I want to get:

\x01\x02\x41

I use this string for redis-cli --pipe.

I don't want python to print the converted string for me.

How could I do it?

now I do like this:

def convert(i:bytes):
    result = list()
    for j in i:
        result.append('\\x%s' % bytes([j]).hex())
    return ''.join(result)

It is too ugly. Is there a better way?

btw.

This is my redis-cli --pipe script:

SET "\xda\xb8\xe6\xc0\x79\x62\xea\x06\xde\x9e\xb0\x4e\x68\xde\x0b\x62" "\x00\x42\xdc\x06\x03\x00\x00\x00"
SET "\x68\xfa\xfc\x03\x36\x99\xb0\x56\xb4\x00\xaf\x61\x42\xe0\x30\x42" "\x01\x42\xdc\x06\x03\x00\x00\x00"
SET "\x40\x98\xb9\x8f\xd8\x4e\x2e\x32\x40\x09\xca\xa4\x55\x52\xde\x7f" "\x02\x42\xdc\x06\x03\x00\x00\x00"
SET "\xd0\x75\xdf\x46\x36\xd0\xb4\xed\xcc\xed\xd6\x27\xf7\xd9\xc5\xa5" "\x03\x42\xdc\x06\x03\x00\x00\x00"

I use it this way:

cat sample.redis | redis-cli --pipe
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Jian Dai
  • 101
  • 1
  • 6
  • https://stackoverflow.com/questions/14674834/php-convert-string-to-hex-and-hex-to-string – Mike May 22 '18 at 15:05
  • @Mike: Why link to that post? Is it because PHP is such a more expressive language than Python? – Martijn Pieters May 22 '18 at 15:08
  • I'm not quite sure why you think you need to produce `\x..` escape sequences for Redis. If you are outputing to a `redis-cli --pipe` command from Python, write the binary data *directly to the pipe* (following the protocol, so `$\r\n` followed by the binary data followed by `\r\n`. The Redis protocol itself doesn't have any `\x..` escape sequence support, and value of the `bytes` object is *not any different* just because the `repr()` of the value uses ASCII characters for bytes that happen to be ASCII codepoints. – Martijn Pieters May 22 '18 at 15:15

1 Answers1

1

When iterating over a bytes value, you get integers; these are trivially converted to hex notation:

def convert(value: bytes):
    return ''.join([f'\\x{b:02x}' for b in value])

Note that this produces a string with literal backslashes, x characters and hexadecimal digit characters. It is no longer a bytes value.

Demo:

>>> print(convert(b'\x01\x02\x41'))
\x01\x02\x41

Just to make sure, you do not need to worry about the bytes value. The repr() representation of a bytes object will always use ASCII characters when the byte value happens to be that of a printable ASCII codepoint. That doesn't mean the value is changed. b'\x01\x02\x41' is equal to b'\x01\x02A'. The Redis protocol doesn't know anything about \x<HH> escape sequences, so don't try to send the above string over the wire.

The escape sequences you produce are bash shell string sequences, and just like Python strings, you don't have to use escapes. As in Python, to Bash the strings "\x01\x02A" and "\x01\x02\x41" have equal values. They only make sense when you are passing in the key and value strings on the command line, not in a text file you pipe to redis-cli.

Moreover, please note that the redis-cli --pipe command takes raw Redis protocol input, not Redis command syntax, see the Redis Mass Insertion documentation. This protocol does not use \xhh sequences, as it does not use shell notation.

Instead, use the following function to generate raw SET commands:

def raw_protocol(cmd: str, *args: bytes):
    return b'\r\n'.join([
        f'*{len(args) + 1}\r\n${len(cmd)}\r\n{cmd}'.encode(),
        *(bv for a in args for bv in (b'$%d' % len(a), a)),
        b''
    ])

For a SET command, use raw_protocol('SET', keybytes, valuebytes) and write out the binary data this produces to a file opened in binary mode.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thank you! I use this in python 3.6: return ''.join(['\\x%02x' % b for b in value]) – Jian Dai May 22 '18 at 15:19
  • @JianDai: the code I posted works perfectly fine in Python 3.6. I'm not sure why you changed it; the `f'...'` formatted string literal syntax is a lot faster than using the older printf-style formatting operation. – Martijn Pieters May 22 '18 at 15:31
  • You are right. I just know, I thought it was a feature in python3.7. :) Study a lot from you, Thank you. – Jian Dai May 22 '18 at 15:34
  • @JianDai: note that I still don't think you are using the redis pipe feature correctly. – Martijn Pieters May 22 '18 at 15:48
  • yes. finally I give up this way. At first I think it will reduce the space in binary key. But Finanally I found redis can compress my txt data better than binary data, and when in unicode, it is easy to use too. – Jian Dai May 31 '18 at 08:45
  • got a dump from redis-cli that was a mish mash of bytes and characters. c# failed me, but python b'/x00U...' totally recognized it this post was super helpful. – James Sep 04 '20 at 13:22