Conversion and evaluation of strings in python

Question

How can I get a Python list from a text file with the following contents?

'hallo'
'hallo\n'
'\x00' * 1
100
'400 + 2'
400 + 2

For example:

ll = ["hallo", "hallo\n", "\x00", 100, 402, 402]

with the types:

[string, string, string, int, int, int]

Means, every string which python understands as int should be from type int.

I tried to use eval but it has difficulties with \n and \x00.

The user input (list of strings to convert) is assumed to be safe.

Answer 1

WARNING : Using eval is dangerous. Be very careful with it, or, better yet, find an alternative without.

That being said, you could define a regex to check if the string looks like something you'd like to eval. For example, anything with only numbers, spaces and mathematical operators could be deemed safe:

import re

l = ['hallo', 'hallo\n', '\x00' * 1, '100', 100, '400 + 2', '400 + - ', 400 + 2]


def string_or_expression(something):
    if isinstance(something, str):
        expression = re.compile('\A[\d\.\-\+\*\/ ]+\Z')
        if expression.match(something):
            try:
                return eval(something)
            except:
                return something
    return something

print([string_or_expression(s) for s in l])
# ['hallo', 'hallo\n', '\x00', 100, 100, 402, '400 + - ', 402]

With Python3, you might use ast.literal_eval, which might be a little less dangerous than a plain eval :

import re
import ast

l = ['hallo', 'hallo\n', '\x00' * 1, '100', 100, '400 + 2', '400 + - ', 400 + 2]


def string_or_expression(something):
    if isinstance(something,str):
      expression = re.compile('\A[\d\.\-\+\*\/ ]+\Z')
      if expression.match(something):
          try:
              return ast.literal_eval(something)
          except:
              return something
    return something

print([string_or_expression(s) for s in l])
# ['hallo', 'hallo\n', '\x00', 100, 100, 402, '400 + - ', 402]

Yet another alternative would be to use @poke's "expression evaluation algorithm", since literal_eval doesn't understand '2 * 3'.

Finally, even a "safe" expression like '2**2**2**2**2**2**2**2**2**2' could bring your server down.

Answer 2

how about:

 def try_eval(x):
    try:
        res=eval(x)
    except:
        res=x
    return res

[try_eval(x) for x in l]

output:

['hallo', 'hallo\n', '\x00', 100, 402]

Answer 3

Let's get serious about avoiding dangerous eval here >:)

import compiler

def is_math(expr):
    """Return True if the expression smells mathematical."""

    try:
        module = compiler.parse(expr)
        stmt, = module.getChildNodes()
        discard, = stmt.getChildNodes()
        code, = discard.getChildNodes()
        return not isinstance(code, compiler.ast.Name)
    except ValueError:
        return False
    except TypeError:
        return False

t = [eval(s) if is_math(s) else s for s in l]

Yes, I took a couple of assumptions here, but you can modify them to suit your needs as strictly as you really need. The AST is pretty easy to understand. When you do a parse, you get a Module. Inside the Module is a Statement. Inside that is (most likely) discard code (that just means it isn't being used anywhere).

If it isn't discard code, we assume it's a string. For one thing, this is likely to prevent any dangerous side effects from eval. (Someone prove me wrong here – wrap a dangerous expression in discard code.)

Inside that is the meat of your expression – from there I assume that anything that is a plain string will appear to be a name in the AST. Anything that isn't a name is probably a number or a math operation.

I think eval should be safe at this point, which is necessary if the expression is really math.