9

Why does the literal evaluation of 5 * 7 fail, while 5 + 7 doesn't?

import ast

print(ast.literal_eval('5 + 7'))
# -> 12

print(ast.literal_eval('5 * 7'))
# -> 
Traceback (most recent call last):
  ...
ValueError: malformed node or string: <_ast.BinOp object at ...>

The documentation doesn't explain this.

I found that problem after answering this question on SO: Getting the result of a string.

tobias_k
  • 81,265
  • 12
  • 120
  • 179
Laurent LAPORTE
  • 21,958
  • 6
  • 58
  • 103
  • 10
    `ast.literal_eval` is less powerful and therefore less dangerous than plain `eval`. I guess that makes it the lesser of two evals... :) – PM 2Ring Nov 14 '16 at 08:27

2 Answers2

16

ast.literal_eval() accepts + in the evaluated data because 5+2j (complex number*) are valid literals. The same applies to -. To keep the code simple, no attempt is made to exclude + or - as a binary operators.

No other operators are allowed; the function is supposed to only accept literals, not expressions.

In other words, that 5 + 7 works is a bug, but one that is hard to fix without breaking support for constructing complex numbers. The implementation limits the use to operands that are numbers, unary + and -, or other binary operators (so you can't use these to concatenate lists or produce a set difference).

Also see several related Python bugtracker entries: #25335 ast.literal_eval fails to parse numbers with leading "+", #22525 ast.literal_eval() doesn't do what the documentation says and #4907 ast.literal_eval does not properly handled complex numbers

* Technically speaking, 2j is a valid literal; Python parses 5+2j as int(5) binop(+) complex(0, 2), and only later produces a complex(5, 2) object from the result, when actually executing the addition.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thank you Martjn, I suspected something like this. Does it means that the AST is the result of the evaluation? I hope it is not. – Laurent LAPORTE Nov 14 '16 at 08:46
  • @LaurentLAPORTE: the function builds an AST from the string, then uses the AST to build the result. So the string is *parsed only*, not directly executed. See the [source code](https://hg.python.org/cpython/file/3.5/Lib/ast.py#l38). – Martijn Pieters Nov 14 '16 at 08:50
7

The question is not "why is * not accepted" but rather "why is + accepted at all".

ast.literal_eval can parse literals, but not expressions. However, in Python, complex numbers are not expressed as a single literal value; instead they consist of the real part and imaginary part added together; the imaginary part is signalled with j. literal_eval thus needs to support binary + and - to support complex number constants such as 1 + 2j or -3.4e-5 - 1.72e9j.

In many versions, including Python 3.5, literal_eval is much more lax than it needs to be - it accepts any chain of additions and subtractions for as long as both the left and right-hand sides evaluate to any number, thus (1 + 3) + 2 + (4 - 5) is still parsed, even if it is not complex constant consisting of real + imaginary part.

+ and - are not accepted unconditionally: if you try to add 2 lists together, it will fail, even though it can parse list literals, and addition is defined for lists:

>>> ast.literal_eval('[1] + [2]')
Traceback (most recent call last):
...
ValueError: malformed node or string: <_ast.BinOp object at 0x7fdddbe785f8>
>>> ast.literal_eval('[1, 2]')
[1, 2]
>>> [1] + [2]
[1, 2]
Antti Haapala -- Слава Україні
  • 129,958
  • 22
  • 279
  • 321