Pythonic way to parse a formatted string and get a dictionary out of it

Question

I have a string that looks like this:

{
    "key A":[
        ["some val", "value a1"],
        ["some val", "value a2"],
        ....................
        ["some val", "value an"]
    ], "key B":[
        ["some val", "value b1"],
        ["some val", "value b2"],
        ...
    ], ...
     , "key X":[
        ["some val", "value x1"],
        ["some val", "value x2"],
        ...
    ]
}

Can anyone tell me what is the proper way in python to get a dictionary out of it where the dictionary should be

{
    "key A": ["val a1", "val a2", ..."val an"],
    "key B": ["val b1", "val b2", ..."val bn"],
    .....
    "key X": ["val x1", "val x2", ..."val xn"],
}

I'm asking this because right now the only code I can think of is a pretty dirty one with a lot of list, splits and replaces and I'm quite sure that this is very wrong way to do it :D. Thanks a lot in advance.

Is this JSON data? It looks like it might be, so you can use: https://stackoverflow.com/questions/4917006/string-to-dictionary-in-python — Alex Riley, Apr 10 '18 at 14:35
@Kajbo I didn't posted my code because, well ... I said "the only code I can think about". I didn't really tried to spend two hours doing a thing that I knew from beginning that is gonna be wrong. — Cristian Harangus, Apr 11 '18 at 06:12
@Alex Riley - thx for the json_loads. I did tried it before asking but for some reason (perhaps bad formatting of my string) didn't managed to make it work. For sure next time that is gonna be my solution. — Cristian Harangus, Apr 11 '18 at 06:33

jpp · Accepted Answer · 2018-04-10T15:22:10.640

4

You can use ast.literal_eval with a dictionary comprehension:

from ast import literal_eval

mystr = """{
    "key A":[
        ["some val", "value a1"],
        ["some val", "value a2"],
        ["some val", "value an"]
    ], "key B":[
        ["some val", "value b1"],
        ["some val", "value b2"],
    ], "key X":[
        ["some val", "value x1"],
        ["some val", "value x2"],
    ]
}"""

res = {k: list(list(zip(*v))[1]) for k, v in literal_eval(mystr).items()}

# {'key A': ['value a1', 'value a2', 'value an'],
#  'key B': ['value b1', 'value b2'],
#  'key X': ['value x1', 'value x2']}

edited Apr 10 '18 at 15:22

answered Apr 10 '18 at 14:36

jpp

159,742
34
281
339

1

I'm downvoting this because eval is naughty generally, and the fact that it appears to be JSON and could come from a remote source increases the security threat. – W4t3randWind Apr 10 '18 at 15:03
2

@W4t3randWind, But I'm using `literal_eval` instead: see [note on scope/security in docs](https://docs.python.org/3/library/ast.html#ast.literal_eval) – jpp Apr 10 '18 at 15:05
2

Excellent, I'll check the docs next time. Removing my downvote. (wish stackoverflow had remindmebot) – W4t3randWind Apr 10 '18 at 15:10

score 3 · Answer 2 · answered Apr 10 '18 at 15:06

As eval is generally regarded as unsafe and python dictionaries are "usually" JSON compatible, as long as all values are also JSON compatible. I would recommend:

import json

mystr = """{
    "key A":[
        ["some val", "value a1"],
        ["some val", "value a2"],
        ["some val", "value an"]
    ], "key B":[
        ["some val", "value b1"],
        ["some val", "value b2"]
    ], "key X":[
        ["some val", "value x1"],
        ["some val", "value x2"]
    ]
}"""

res = json.loads(mystr)

for dealing with trailing commas:

import json
from jsoncomment import JsonComment


mystr = """{
    "key A":[
        ["some val", "value a1"],
        ["some val", "value a2"],
        ["some val", "value an"]
    ], "key B":[
        ["some val", "value b1"],
        ["some val", "value b2"],
    ], "key X":[
        ["some val", "value x1"],
        ["some val", "value x2"],
    ]
}"""

res = JsonComment(json).loads(mystr)

docs for JsonComment: https://pypi.python.org/pypi/jsoncomment

ref: Can json.loads ignore trailing commas?

dawg · Answer 3 · 2018-04-11T23:38:10.043

As others have pointed out, the issue is the non-standard trailing comma at the end of the list elements of the json string.

You can use ast.literal_eval() in the example.

However, if you need to write your own json parser to deal with json that the Python library parser does not handle, you can use PyParsing to do so.

An example JSON parser, written in PyParsing, can easily be adapted to handle json with optional trailing commas:

testdata='''\
{
    "key A":[
        ["some val", "value a1"],
        ["some val", "value a2"],
        ["some val", "value an"],
    ], "key B":[
        ["some val", "value b1"],
        ["some val", "value b2"]
    ],
    "key X":[
        ["some val", "value x1"],
        ["some val", "value x2"]
    ]
}'''

json_bnf = """
object 
    { members } 
    {} 
members 
    string : value 
    members , string : value 
array 
    [ elements ]
    [] 
elements 
    value 
    elements , value 
value 
    string
    number
    object
    array
    true
    false
    null
"""

from pyparsing import *
import ast

def make_keyword(kwd_str, kwd_value):
    return Keyword(kwd_str).setParseAction(replaceWith(kwd_value))
TRUE  = make_keyword("true", True)
FALSE = make_keyword("false", False)
NULL  = make_keyword("null", None)

LBRACK, RBRACK, LBRACE, RBRACE, COLON = map(Suppress, "[]{}:")

jsonString = dblQuotedString().setParseAction(removeQuotes)
jsonNumber = pyparsing_common.number()

jsonObject = Forward()
jsonValue = Forward()
jsonElements = delimitedList( jsonValue )
jsonArray = Group(LBRACK + Optional(jsonElements, []) + Optional(Suppress(",")) + RBRACK)
jsonValue << (jsonString | jsonNumber | Group(jsonObject)  | jsonArray | TRUE | FALSE | NULL)
memberDef = Group(jsonString + COLON + jsonValue)
jsonMembers = delimitedList(memberDef)
jsonObject << Dict(LBRACE + Optional(jsonMembers) + RBRACE)

jsonComment = cppStyleComment 
jsonObject.ignore(jsonComment)

And the result is the same as ast parsing it:

>>> dict(results.asList())==ast.literal_eval(testdata)
True

The advantage of 'rolling your own' in this case is to have control of what non-standard elements you have and how you want to handle them.

(Thanks to Paul McGuire for the PyParsing module and the json parser...)

Renan Quirino · Answer 4 · 2018-04-10T14:47:42.370

0

You could do this:

your_dictionary = eval(original_str)     

for key, values in your_dictionary.items():
    v = list()
    for value in values:
        v += value
    your_dictionary[key] = v

edited Apr 10 '18 at 14:47

answered Apr 10 '18 at 14:42

Renan Quirino

83
1
7

Do not use `eval` with its inherent risk when a far less risky alternative is available with `ast.literal_eval`. -1 – dawg Apr 11 '18 at 19:29

Pythonic way to parse a formatted string and get a dictionary out of it

4 Answers4