Way to use ast.literal_eval() to convert string into a datetime?

Question

I have a string "{'datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)}" and I want to convert this to the object it represents. Using ast.literal_eval() gives:

ValueError: malformed string; 

because it does not allow for construction of objects (i.e. the datetime call). Is there anyway to either get ast to process this correctly, or to secure eval to prevent code injection?

Answer 1

Following up on Ignacio Vazquez-Abrams' idea:

import ast
import datetime

def parse_datetime_dict(astr,debug=False):
    try: tree=ast.parse(astr)
    except SyntaxError: raise ValueError(astr)
    for node in ast.walk(tree):
        if isinstance(node,(ast.Module,ast.Expr,ast.Dict,ast.Str,
                            ast.Attribute,ast.Num,ast.Name,ast.Load, ast.Tuple)): continue
        if (isinstance(node,ast.Call)
                and isinstance(node.func, ast.Attribute)
                and node.func.attr == 'datetime'): continue
        if debug:
            attrs=[attr for attr in dir(node) if not attr.startswith('__')]
            print(node)
            for attrname in attrs:
                print('    {k} ==> {v}'.format(k=attrname,v=getattr(node,attrname)))
        raise ValueError(astr)
    return eval(astr)

good_strings=["{'the_datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)}"]
bad_strings=["__import__('os'); os.unlink",
             "import os; os.unlink",
             "import(os)", # SyntaxError
             ]

for astr in good_strings:
    result=parse_datetime_dict(astr)    
    print('{s} ... [PASSED]'.format(s=astr))

for astr in bad_strings:
    try:
        result=parse_datetime_dict(astr)
    except ValueError:
        print('{s} ... [REJECTED]'.format(s=astr))
    else:
        sys.exit('ERROR: failed to catch {s!r}'.format(s=astr))

yields

{'the_datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)} ... [PASSED]
__import__('os'); os.unlink ... [REJECTED]
import os; os.unlink ... [REJECTED]
import(os) ... [REJECTED]

Answer 2

You could extract the (2010, 11, 21, 0, 56, 58) characters from the string using a regex, pass that to ast.literal_eval() to get a tuple, and then pass that tuple to datetime.datetime(*that_tuple) to get the object. Sounds like a lot, but each of the steps is very simple (and secure). Here's what I'm talking about:

import ast
import datetime
import re

s = "{'datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)}"
m = re.search(r"datetime(\((\d+)(,\s*\d+)*\))", s)
if m:  # Match?
    args = ast.literal_eval(m.group(1))
    print(datetime.datetime(*args))  # -> 2010-11-21 00:56:58

This searches for the pattern datetime(<comma separated list of integers>) in the string and passes just the list of literal integer values to ast.literal_eval() for conversion to a tuple — which should always succeed and is code-injection resistant. Using a technique like this to defend against injection attacks is called "Context-Sensitive String Evaluation" or CSSE^†.

---

† Pietraszek, Tadeusz and Chris Vanden Berghe, "Defending Against Injection Attacks Through Context-Sensitive String Evaluation", Conference Paper, 2005

Answer 3

Instead of writing lots of code, do not use ast when you have to parse datetime objs. You can run eval(). BTW please note that you may have security issues using this function if the string can contain dodgy python commands.

Here is how it works:

>>> x="{'datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)}"
>>> b=eval(x)
>>> b
{'datetime': datetime.datetime(2010, 11, 21, 0, 56, 58)}
>>> b["datetime"].year
2010

Enjoy! :D

Answer 4

Use the language services to compile it into an AST, walk the AST making sure that it contains only whitelisted node sets, then execute it.

Answer 5

I ran into this issue and solved it by replacing the datetime object with a string like this:

            if mystrobj.__contains__('datetime.datetime('):
                mystrobj = re.sub(r"datetime.datetime([(),0-9 ]*),", r"'\1',", mystrobj)

Answer 6

Use repr(datetime.datetime.utcnow()) to save in your dict or your file. Look,

>>> import datetime
>>> oarg = datetime.datetime.utcnow()
>>> oarg
datetime.datetime(2013, 2, 6, 12, 39, 51, 709024)
>>> butiwantthis = repr(oarg)
>>> butiwantthis
'datetime.datetime(2013, 2, 6, 12, 39, 51, 709024)'
>>> eval(oarg)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: eval() arg 1 must be a string, bytes or code object
>>> eval(butiwantthis)
datetime.datetime(2013, 2, 6, 12, 39, 51, 709024)

Cool! eval() works!

Take care with import

>>> from datetime import datetime, date, time
>>> oarg = datetime.datetime.utcnow()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'datetime.datetime' has no attribute 'datetime'
>>> oarg = datetime.utcnow()
>>> oarg
datetime.datetime(2013, 2, 6, 12, 41, 51, 870458)
>>> butiwantthis = repr(oarg)
>>> butiwantthis
'datetime.datetime(2013, 2, 6, 12, 41, 51, 870458)'
>>> eval(butiwantthis)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
AttributeError: type object 'datetime.datetime' has no attribute 'datetime'
>>> # wrong import

Perfect!

Ps: Python3.3