Unicode decode bytes error (Python)

Question

What does this error message mean?

SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 123-125: truncated \uXXXX escape

I get this error reported at a position in the comments, which contain only non-Unicode chars.

The problematic code is the following:

""" loads Font combinations from a file
#
# The font combinations are saved in the format:
% -> Palatino, Helvetica, Courier
\usepackage{mathpazo}                 %% --- Palatino (incl math)
\usepackage[scaled=.95]{helvet}       %% --- Helvetica (Arial)
\usepackage{courier}                  %% --- Courier
\renewcommand{\fontdesc}{Palatino, Arial, Courier}
% <-------------------
#
# with "% ->" indicating the start of a new group
# and "% <" indicating the end.
"""

I wanted to include the code as well, but stackoverflow refused to except the question with the code because of a to low quality question ... — Matthias Pospiech, May 26 '13 at 15:58
The code the that is problematic is a comment, nothing that even shall be executed. In that sense it is different. — Matthias Pospiech, May 26 '13 at 16:48
Another good question is why you are using 'unicodeescape' as the encoding for the file. I think that's the fundamental error. UTF-8 is likely to be a better choice. — Lennart Regebro, May 26 '13 at 20:04

score 2 · Answer 1 · answered May 26 '13 at 17:16

As the others have said, it's trying to parse \usepackage as a Unicode escape and failing because it's invalid. The way around this is to escape the backslash:

"""\\usepackage""

Or to use a raw string instead:

r"""\usepackage"""

PEP 257, which covers docstring conventions, suggests the latter.

score 2 · Accepted Answer · answered May 26 '13 at 17:35

2

It's worth noting that the "problematic code" is not technically a comment, but a multiline string which will be evaluated during bytecode compilation.

Depending in its location in the source file, it may end up in a docstring, so it has to be syntactically valid.

For example...

>>> def myfunc():
...     """This is a docstring."""
...     pass
>>> myfunc.__doc__
'This is a docstring.'
>>> help(myfunc)
Help on function myfunc in module __main__:

myfunc()
    This is a docstring.

There's no true multiline comment delimiter in Python, so if you don't want it to be evaluated, use several single-line comments...

# This is my comment line 1
# ...line 2
# etc.
def myfunc():
    pass

answered May 26 '13 at 17:35

Aya

39,884
6
55
55

or raw string literals: `r"\uxxxxx"` – jfs May 26 '13 at 17:37
@J.F.Sebastian Well, it sounds like the OP doesn't actually want to use docstrings, so I figured it'd be simplest just to use regular single-line comments, since that's the only way to prevent Python from attempting to evaluate the 'comment' block. I guess I could add that in, although it's already mentioned in Cairnarvon's answer. – Aya May 26 '13 at 17:44

score 1 · Answer 3 · answered May 26 '13 at 16:06

1

It means that the \uXXXX escape sequence in the data you are decoding is invalid. Specifically it means it's to short. Most likely you have the text '\U' somewhere in the text, but not followed by a Unicode character number.

answered May 26 '13 at 16:06

Lennart Regebro

167,292
41
224
251

Yes, my comment includes the line `\usepackage{mathpazo} %% --- Palantino (incl math)`. But why is a comment parsed ? – Matthias Pospiech May 26 '13 at 16:45
@MatthiasPospiech: a docstring is not a comment. – Cairnarvon May 26 '13 at 17:20
Ok, I have no idea what a docstring is, but that makes it clear why the error is raised. – Matthias Pospiech May 26 '13 at 17:21
@MatthiasPospiech: I assume decoding is done before parsing in any case, so I wouldn't be surprised if comments give the same error. – Lennart Regebro May 26 '13 at 20:06

score 1 · Answer 4 · answered May 26 '13 at 17:10

Python 3 strings are Unicode, so it attempts to decode the '\u' escapes. So, even though you are trying to use a string as a comment, it will still attempt to decode it.

An actual comment, such as:

#\usepackage{mathpazo}

will not be decoded.

If you notice it's in the class of SyntaxErrors, which means that even if it's 'unreachable code', it raises a flag.

Unicode decode bytes error (Python)

4 Answers4