Optimize byte array escaping performance python

Question

I need to perform custom escaping over a byte array in python. However, during escaping python converts bytes to integers, making performance optimization very difficult. How can I speed up my escaping function?

ESCAPE_DICT={
    0x00: [0x5C,0x7A], # null        -> \z 0x5c 0x7a
    0x22: [0x5C,0x71], # "           -> \q 0x5c 0x71
    0x3B: [0x5C,0x73], # ;           -> \s 0x5c 0x73
    0x5C: [0x5C,0x5C], # \           -> \\ 0x5c 0x5c
    0x0A: [0x5C,0x6E], # line-feed   -> \n 0x5c 0x6e
    0x0C: [0x5C,0x66], # form-feed   -> \f 0x5c 0x66
    0x0D: [0x5C,0x63], # carr-return -> \c 0x5c 0x63
}

def escape(string: bytes):
    str_len=string.__len__()
    escaped_list=[]
    for i in range(0,str_len):
        curr_byte=string[i]
        escape = ESCAPE_DICT.get(curr_byte)
        if escape is None:
            # Don't escape current byte
            escaped_list.append(curr_byte)
        else:
            # Escape current byte
            escaped_list.extend(escape)
    return bytes(escaped_array)

maybe useful: https://stackoverflow.com/questions/6116978/how-to-replace-multiple-substrings-of-a-string — Walter Tross, Oct 13 '21 at 19:39
The problem is that I have to perform a cycle over the string N times where N is the number of possible escaping patterns — AndreaCostanzo1, Oct 13 '21 at 19:59
I have a different implementation, but this also looks right. Can you provide a test case where it doesn't work? Why would you need to loop over escape patterns? — Kenny Ostrom, Oct 13 '21 at 20:32
@KennyOstrom the problem is not that it doesn't work. But here I'm looking to improve the performance of my algorithm. — AndreaCostanzo1, Oct 13 '21 at 20:41

Walter Tross · Accepted Answer · 2021-10-14T10:11:08.680

import re

ESCAPE_DICT = {
    b'\x00': rb'\z',  # null
    b'"': rb'\q',
    b';': rb'\s',
    b'\\': rb'\\',
    b'\n': rb'\n',  # linefeed
    b'\f': rb'\f',  # formfeed
    b'\r': rb'\c',  # carriage return
}
ESCAPE_CLASS = '[' + ''.join(r'\x' + e.hex() for e in ESCAPE_DICT) + ']'
ESCAPE_REGEX = re.compile(ESCAPE_CLASS.encode())


def escape(string: bytes) -> bytes:
    return re.sub(ESCAPE_REGEX, lambda m: ESCAPE_DICT[m.group(0)], string)


x = b'"abc\ndef\rpqr\x00stu\\xyz"'
y = escape(x)

from pprint import pprint
pprint(ESCAPE_CLASS)
pprint(ESCAPE_REGEX)
pprint(x)
pprint(y)
# =>
# '[\\x00\\x22\\x3b\\x5c\\x0a\\x0c\\x0d]'
# re.compile(b'[\\x00\\x22\\x3b\\x5c\\x0a\\x0c\\x0d]')
# b'"abc\ndef\rpqr\x00stu\\xyz"'
# b'\\qabc\\ndef\\cpqr\\zstu\\\\xyz\\q'

You can read the rb prefix as “raw bytes”.

Your escapes are a bit strange, though. E.g., the carriage return is normally \r, not \c, and \s normally stands for generic whitespace.

Using regex improved escaping latency by 200us. Thanks walter! — AndreaCostanzo1, Oct 14 '21 at 09:54
You're welcome. Do you mind if I add the regex tag for ease of finding? — Walter Tross, Oct 14 '21 at 09:59

Optimize byte array escaping performance python

1 Answers1