Converting unicode sequence to string in Python3 but allow paths in string

Question

There is at least one related question on SO that proved useful when trying to decode unicode sequences.

I am preprocessing a lot of texts with a lot of different genres. Some are economical, some are technical, and so on. One of the caveats is converting unicode sequences:

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojt\u0115ch \u010camek.

Such a string needs to be converted to actual characters:

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojtĕch Čamek.

which can be done like this:

s = "'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojt\u0115ch \u010camek."
s = s.encode('utf-8').decode('unicode-escape')

(At least this works when s is an input line taken from a utf-8 encoded text file. I can't seem to get this to work on an online service like REPL.it, where the output is encoded/decoded differently.)

In most cases, this works fine. However, when directory structure paths are seen in the input string (often the case for technical documents in my data set) then UnicodeDecodeErrors occur.

Given the following data unicode.txt:

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojt\u0115ch \u010camek, Financial Director and Director of Controlling.
Voor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:\udfs\math.dll (op Windows)).

With bytestring representation of:

b"'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojt\\u0115ch \\u010camek, Financial Director and Director of Controlling.\r\nVoor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:\\udfs\\math.dll (op Windows))."

The following script will fail when decoding the second line in the input file:

with open('unicode.txt', 'r', encoding='utf-8') as fin, open('unicode-out.txt', 'w', encoding='utf-8') as fout:
    lines = ''.join(fin.readlines())
    lines = lines.encode('utf-8').decode('unicode-escape')

    fout.write(lines)

With trace:

Traceback (most recent call last):
  File "C:/Python/files/fast_aligning/unicode-encoding.py", line 3, in <module>
    lines = lines.encode('utf-8').decode('unicode-escape')
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 275-278: truncated \uXXXX escape

Process finished with exit code 1

How can I ensure that the first sentence is still 'translated' correctly, as shown before, but that the second one remains untouched? Expected output for the two lines given would thus be as follows, where the first line has changed and the second hasn't.

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojtĕch Čamek.
Voor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:\udfs\math.dll (op Windows)).

Answer 1

The raw_unicode_escape codec in the ignore mode seems to do the trick. I'm inlining the input as a raw byte longstring here, which should by my reasoning be equivalent to reading it from a binary file.

input = br"""
'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojt\u0115ch \u010camek, Financial Director and Director of Controlling.
Voor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:\udfs\math.dll (op Windows)).
"""

print(input.decode('raw_unicode_escape', 'ignore'))

outputs

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojtĕch Čamek, Financial Director and Director of Controlling.
Voor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:s\math.dll (op Windows)).

Note that the \udf in d:\udfs gets mangled, as the codec attempts to start reading an \uXXXX sequence, but gives up at the s.

An alternative (likely slower) would be to use a regexp to find the valid Unicode sequences within decoded data. This assumes .decode()ing the full input string as UTF-8 is possible, though. (The .encode().decode() dance is necessary since strings can't be encoded, just bytes. One could also use chr(int(m.group(0)[2:], 16)).)

escape_re = re.compile(r'\\u[0-9a-f]{4}')
output = escape_re.sub(lambda m: m.group(0).encode().decode('unicode_escape'), input.decode()))

outputs

'Korado's output has gone up from 180,000 radiators per year to almost 1.7 million today,' says Vojtĕch Čamek, Financial Director and Director of Controlling.
Voor alle bestanden kan de naam met de volledige padnaam (bijvoorbeeld: /u/slick/udfs/math.a (op UNIX), d:\udfs\math.dll (op Windows)).

Since \udf doesn't have 4 hexadecimal characters, the d:\udfs path is spared here.

Answer 2

The input is ambiguous. The right answer does not exist in the general case. We could use heuristics that produce an output that looks right most of the time e.g., we could use a rule such as "if \uxxxx sequence (6 chars) is a part of an existing path then don't interpret it as a Unicode escape" and the same for \Uxxxxxxxx (10 chars) sequences e.g., an input that is similar to the one from the question: b"c:\\U0001f60f\\math.dll" can be interpreted differently depending on whether c:\U0001f60f\math.dll file actually exists on the disk:

#!/usr/bin/env python3
import re
from pathlib import Path


def decode_unicode_escape_if_path_doesnt_exist(m):
    path = m.group(0)
    return path if Path(path).exists() else replace_unicode_escapes(path)


def replace_unicode_escapes(text):
    return re.sub(
        fr"{unicode_escape}+",
        lambda m: m.group(0).encode("latin-1").decode("raw-unicode-escape"),
        text,
    )


input_text = Path('broken.txt').read_text(encoding='ascii')
hex = "[0-9a-fA-F]"
unicode_escape = fr"(?:\\u{hex}{{4}}|\\U{hex}{{8}})"
drive_letter = "[a-zA-Z]"
print(
    re.sub(
        fr"{drive_letter}:\S*{unicode_escape}\S*",
        decode_unicode_escape_if_path_doesnt_exist,
        input_text,
    )
)

Specify the actual encoding of your broken.txt file in the read_text() if there are non-ascii characters in the encoded text.

What specific regex to use to extract paths depends on the type of input that you get.

You could complicate the code by trying to substitute one possible Unicode sequence at a time (the number of replacements grows exponentially with the number of candidates in this case e.g., if there are 10 possible Unicode escape sequences in a path then there are 2**10 decoded paths to try).

Answer 3

I had already written this code when AKX posted his answer. I still think it applies.

The idea is to capture unicode sequence candidates with a regex (and try to exclude paths, e.g. parts that are preceded with any letter and a colon (e.g. c:\udfff). If decoding does fail, we'll return the original string.

with open('unicode.txt', 'r', encoding='utf-8') as fin, open('unicode-out.txt', 'w', encoding='utf-8') as fout:
    lines = ''.join(fin.readlines())
    lines = lines.strip()
    lines = unicode_replace(lines)
    fout.write(lines)


def unicode_replace(s):
    # Directory paths in a text are seen as unicode sequences but will fail to decode, e.g. d:\udfs\math.dll
    # In case of such failure, we'll pass on these sentences - we don't try to decode them but leave them
    # as-is. Note that this may leave some unicode sequences alive in your text.
    def repl(match):
        match = match.group()
        try:
            return match.encode('utf-8').decode('unicode-escape')
        except UnicodeDecodeError:
            return match

    return re.sub(r'(?<!\b[a-zA-Z]:)(\\u[0-9A-Fa-f]{4})', repl, s)