16

I am converting 2 MB of data as a string into a dict. The input is serialized in JSON.

Anyways I am currently using ast.literal_eval and I get the dictionary I want, but then when I tried just running eval it seems to run faster, and also returns the same result.

Is there any reason to use the ast module or the json module when eval works just fine?

MxLDevs
  • 19,048
  • 36
  • 123
  • 194
  • If you want to distribute the dictionary alongside with your code, you can laso simply put it in a Python module and import it. That way, you make clear this is Python code. – Sven Marnach Mar 30 '12 at 19:57
  • 1
    eval is prone to security threats. Use only when you absolutely control what gets eval'ed – jldupont Mar 30 '12 at 19:49

4 Answers4

34

I don't really like this attitude on stackoverflow (and elsewhere) telling people without any context that what they are doing is insecure and they shouldn't do it. Maybe it's just a throwaway script to import some data, in that case why not choose the fastest or most convenient way?

In this case, however, json.loads is not only more secure, but also more than 4x faster (depending on your data). 

In [1]: %timeit json.loads(data)
10000 loops, best of 3: 41.6 µs per loop

In [2]: %timeit eval(data)
10000 loops, best of 3: 194 µs per loop

In [3]: %timeit ast.literal_eval(data)
1000 loops, best of 3: 269 µs per loop

If you think about it makes sense json is a such more constrained language/format than python, so it must be faster to parse with an optimized parser.

Maarten
  • 4,549
  • 4
  • 31
  • 36
25

Yes, there's definitely a reason: eval() is evil. Your code might read untrusted data one day, an this would allow an attacker to run arbitrary code on your machine.

You shouldn't use ast.literal_eval() to decode JSON either. It cannot decode every valid JSON string and is not meant to be used for this purpose. Simply use json.loads(), it's reasonably fast.

Sven Marnach
  • 574,206
  • 118
  • 941
  • 841
  • 1
    What if they can only run it on their own machine? (eg: they download it and run it) – MxLDevs Mar 30 '12 at 19:51
  • 2
    @Keikoku: So if it's not your machine, you don't care? – Sven Marnach Mar 30 '12 at 19:53
  • I guess they could spread it around and do dubious things with it and credit me for the work. – MxLDevs Mar 30 '12 at 19:54
  • 1
    what about defining pyson, an amazing python/python interchange/storage format, whose only spec is : anything parseable without error by `ast.literal_eval()` :) – yota May 15 '14 at 14:33
16

No. Unless you hit one of two scenarios:

  1. That's not JSON!

    Someone puts __import__('os').system('rm -rf /') in the file instead. You are boned.

  2. It's JSON, but not the Python-like part!

    Someone puts true, false, null, or a Unicode escape somewhere in it. Happy birthday.

Community
  • 1
  • 1
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • How about `eval(json_str, {'false': False, 'true': True, 'null': None, '__builtins__': {}})`? ;-) – Nas Banov Jun 03 '12 at 11:29
  • 3
    @NasBanov: see http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html – Janus Troelsen Nov 26 '12 at 02:01
  • @JanusTroelsen, yeah - see this http://stackoverflow.com/questions/7282905/does-converting-json-to-dict-with-eval-a-good-choice/7282959#comment14167166_7282959 - seems related by time too – Nas Banov Nov 26 '12 at 22:42
  • @NasBanov: and now try `json_str = '"\\u2603"'`. The *expected* output is `u'\u2603'`. What you'll get instead is `'\\u2603'`. You cannot map your way out of that one either. – Martijn Pieters Jan 22 '15 at 18:57
2

Not an answer exactly, but it should be noted that eval and literal_eval are not the same thing. The ast.literal_eval won't run arbitrary code.

That said, I agree with using JSON; I just wanted to point out that eval != literal_eval

Induane
  • 1,774
  • 2
  • 13
  • 11