2

I'm trying to use ijson instead of json to be able to efficiently dump/load dictionaries to/from strings (in-memory, not from a file) [1].

Are there any examples for ijson analogous to standard dumping/loading with json? All sources I've seen that use ijson have examples with files only.

[1] -- Dictionaries are being converted to strings in order to store them in a set

ballade4op52
  • 2,142
  • 5
  • 27
  • 42

2 Answers2

1

You can reach both goals (dict to string and string to dict without using the json lib).

Let's look at the above example:

import ast
di = {"a":2, "b": 3}

print (str(di))
>>> "{'a': 2, 'b': 3}"

print (type(str(di)))
>>> <class 'str'>

print (ast.literal_eval(str(di)))
>>> {'a': 2, 'b': 3}

print (type(ast.literal_eval(str(di))))
>>> <class 'dict'>

You have a dictionary, In order to turn it into string, you just need to cast it into str.

If you want to return the str to dict you can use the ast lib.

You can read more about ast.literal_eval

ast.literal_eval(node_or_string)

Safely evaluate an expression node or a Unicode or Latin-1 encoded string containing a Python literal or container display.

EDIT:

After further investigation and following this SO question It seems that using json would get you to a faster results (performance).

Look at abarnert answer at the attached link, specifacally on:

Why is json.loads so much faster ?

Because Python literal syntax is a more complex and powerful language than JSON, it's likely to be slower to parse. And, probably more importantly, because Python literal syntax is not intended to be used as a data interchange format (in fact, it's specifically not supposed to be used for that), nobody is likely to put much effort into making it fast for data interchange.

import json
di = {"a":2, "b": 3}

print (json.dumps(di))
>>> '{"a": 2, "b": 3}'

print (type(json.dumps(di)))
>>> <class 'str'>

print (json.loads(json.dumps(di)))
>>> {'a': 2, 'b': 3}

print (type(json.loads(json.dumps(di))))
>>> <class 'dict'>
omri_saadon
  • 10,193
  • 7
  • 33
  • 58
  • I was previously considering eval, but I noticed that it was associated with poor performance. The reason I thought ijson would be a good approach is due to its efficiency. However, it's efficient relative to standard json so I don't know how it compares to eval. – ballade4op52 Jul 05 '17 at 07:07
  • 1
    read this [stackoverflow answer - Using python's eval() vs. ast.literal_eval()?](https://stackoverflow.com/questions/15197673/using-pythons-eval-vs-ast-literal-eval) in order to get more idea. – omri_saadon Jul 05 '17 at 07:09
  • I could be more clear. By efficiency, I mean speed. Let me know if you have any thoughts on what would be the most speed-efficient method of dict/str conversion. – ballade4op52 Jul 05 '17 at 07:11
  • 1
    @Phillip , Read this [answer - Why is json.loads an order of magnitude faster than ast.literal_eval?](https://stackoverflow.com/questions/21223392/why-is-json-loads-an-order-of-magnitude-faster-than-ast-literal-eval) in order to understand why using json is much faster than literal_eval – omri_saadon Jul 05 '17 at 07:16
  • @Phillip , I've added an EDIT section to my answer, still all is in-memory. Good luck ! – omri_saadon Jul 05 '17 at 07:27
1

To complement omri_saadon's answer in the vein of performance,

Here is a link (serializer speed comparisons) that measures some lesser known JSON modules, one of which was found to effectively outperform the others (msgpack) and can be used interchangeably with json.

Hence, instead of ijson or standard json, msgpack is presently being used for the dict/str conversions.

To install: pip install msgpack

To use: import msgpack as json # can use dumps/loads functions

ballade4op52
  • 2,142
  • 5
  • 27
  • 42
  • 1
    As of [`msgpack-python == 0.5.0`](https://pypi.org/project/msgpack-python/0.5.0/) has been renamed to [`msgpack`](https://pypi.python.org/pypi/msgpack/0.5.1), and deprecated as of [`msgpack-python == 0.5.1`](https://pypi.org/project/msgpack-python/0.5.1/). – 0 _ Jan 27 '18 at 01:53
  • 1
    An additional benchmark: http://lpetr.org/blog/archives/faster-json-parsing-python-ijson – 0 _ Jan 27 '18 at 01:57