XPath like query for nested python dictionaries

Question

Is there a way to define a XPath type query for nested python dictionaries.

Something like this:

foo = {
  'spam':'eggs',
  'morefoo': {
               'bar':'soap',
               'morebar': {'bacon' : 'foobar'}
              }
   }

print( foo.select("/morefoo/morebar") )

>> {'bacon' : 'foobar'}

I also needed to select nested lists ;)

This can be done easily with @jellybean's solution:

def xpath_get(mydict, path):
    elem = mydict
    try:
        for x in path.strip("/").split("/"):
            try:
                x = int(x)
                elem = elem[x]
            except ValueError:
                elem = elem.get(x)
    except:
        pass

    return elem

foo = {
  'spam':'eggs',
  'morefoo': [{
               'bar':'soap',
               'morebar': {
                           'bacon' : {
                                       'bla':'balbla'
                                     }
                           }
              },
              'bla'
              ]
   }

print xpath_get(foo, "/morefoo/0/morebar/bacon")

[EDIT 2016] This question and the accepted answer are ancient. The newer answers may do the job better than the original answer. However I did not test them so I won't change the accepted answer.

@MarcoS It would be more interesting with lists where the path microlanguage would return multiple items. — Pavel Šimerda, Oct 14 '17 at 09:44
@PavelŠimerda Yes, way more interesting, especially with wildcard queries (find all values under a specific key), and then - also recurse down lists or [named]tuples... — Tomasz Gandor, Apr 25 '18 at 21:36
This question (in Python) essentially asks for a recommendation of a 3rd party library. — user7610, Mar 01 '21 at 08:35

score 22 · Answer 1 · answered Sep 26 '14 at 01:34

One of the best libraries I've been able to identify, which, in addition, is very actively developed, is an extracted project from boto: JMESPath. It has a very powerful syntax of doing things that would normally take pages of code to express.

Here are some examples:

search('foo | bar', {"foo": {"bar": "baz"}}) -> "baz"
search('foo[*].bar | [0]', {
    "foo": [{"bar": ["first1", "second1"]},
            {"bar": ["first2", "second2"]}]}) -> ["first1", "second1"]
search('foo | [0]', {"foo": [0, 1, 2]}) -> [0]

but this does not allow to modify the dict :( – Gaetan Nov 22 '18 at 13:55 — Gaetan, Nov 22 '18 at 13:55

score 19 · Answer 2 · answered May 12 '13 at 13:53

There is an easier way to do this now.

http://github.com/akesterson/dpath-python

$ easy_install dpath
>>> dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar")

... done. Or if you don't like getting your results back in a view (merged dictionary that retains the paths), yield them instead:

$ easy_install dpath
>>> for (path, value) in dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar", yielded=True)

... and done. 'value' will hold {'bacon': 'foobar'} in that case.

The iterated statement doesn't run---there's no body to the for statement. — Mittenchops, Jul 02 '13 at 15:04

score 18 · Accepted Answer · answered Sep 06 '11 at 13:25

18

Not exactly beautiful, but you might use sth like

def xpath_get(mydict, path):
    elem = mydict
    try:
        for x in path.strip("/").split("/"):
            elem = elem.get(x)
    except:
        pass

    return elem

This doesn't support xpath stuff like indices, of course ... not to mention the / key trap unutbu indicated.

answered Sep 06 '11 at 13:25

Johannes Charra

29,455
6
42
51

In 2011 maybe there weren't as many options as there are today, but in 2014, I think, solving the problem this way is not elegant and should be avoided. – nikolay Sep 26 '14 at 01:37
11

@nikolay is this just a guess or are there solutions that solve this more nicely? – Nils Werner Dec 16 '15 at 12:23

ankostis · Answer 4 · 2014-01-01T11:29:03.500

14

There is the newer jsonpath-rw library supporting a JSONPATH syntax but for python dictionaries and arrays, as you wished.

So your 1st example becomes:

from jsonpath_rw import parse

print( parse('$.morefoo.morebar').find(foo) )

And the 2nd:

print( parse("$.morefoo[0].morebar.bacon").find(foo) )

PS: An alternative simpler library also supporting dictionaries is python-json-pointer with a more XPath-like syntax.

edited Jan 01 '14 at 11:29

answered Jan 01 '14 at 10:22

ankostis

8,579
3
47
61

1

Note that jsonpath uses eval and jsonpath-rw looks unmaintained (it also says some features are missing, but I haven't tried it). – Sam Brightman Aug 19 '17 at 10:53

score 10 · Answer 5 · edited Oct 07 '21 at 08:14

dict > jmespath

You can use JMESPath which is a query language for JSON, and which has a python implementation.

import jmespath # pip install jmespath

data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}

jmespath.search('root.section.item2', data)
Out[42]: 'value2'

The jmespath query syntax and live examples: http://jmespath.org/tutorial.html

dict > xml > xpath

Another option would be converting your dictionaries to XML using something like dicttoxml and then use regular XPath expressions e.g. via lxml or whatever other library you prefer.

from dicttoxml import dicttoxml  # pip install dicttoxml
from lxml import etree  # pip install lxml

data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
xml_data = dicttoxml(data, attr_type=False)
Out[43]: b'<?xml version="1.0" encoding="UTF-8" ?><root><root><section><item1>value1</item1><item2>value2</item2></section></root></root>'

tree = etree.fromstring(xml_data)
tree.xpath('//item2/text()')
Out[44]: ['value2']

Json Pointer

Yet another option is Json Pointer which is an IETF spec that has a python implementation:

https://github.com/stefankoegl/python-json-pointer

From the jsonpointer-python tutorial:

from jsonpointer import resolve_pointer

obj = {"foo": {"anArray": [ {"prop": 44}], "another prop": {"baz": "A string" }}}

resolve_pointer(obj, '') == obj
# True

resolve_pointer(obj, '/foo/another%20prop/baz') == obj['foo']['another prop']['baz']
# True

>>> resolve_pointer(obj, '/foo/anArray/0') == obj['foo']['anArray'][0]
# True

checking this, as I wouldn't want to change backend API, but to traverse the output json — swdev, Jun 11 '20 at 11:48
Converting from dict to xml and than using path doesn't seem to me as good practise. — Miroslav Karpíšek, Nov 19 '21 at 15:53

score 5 · Answer 6 · answered Feb 01 '18 at 17:48

5

If terseness is your fancy:

def xpath(root, path, sch='/'):
    return reduce(lambda acc, nxt: acc[nxt],
                  [int(x) if x.isdigit() else x for x in path.split(sch)],
                  root)

Of course, if you only have dicts, then it's simpler:

def xpath(root, path, sch='/'):
    return reduce(lambda acc, nxt: acc[nxt],
                  path.split(sch),
                  root)

Good luck finding any errors in your path spec tho ;-)

answered Feb 01 '18 at 17:48

d1zzyg

51
2
4

This will avoid converting things to ints if a node is a dict: def xpath(root, path, sep='/'): return reduce(lambda node, key: node[key if hasattr(node, 'keys') else int(key)], path.split(sep), root) – samwyse Mar 28 '18 at 17:18
Cool solution. For Python 3, need `from functools import reduce` though. – Adrian W Jul 11 '18 at 15:25
I like this *terseness* - the parser should give a `key not found` error when the path spec is wrong, so that should not be very painful to debug. – michaPau Apr 20 '19 at 18:28
great solution, but breaks when you have a dictionary key as an integer, e.g. in `d1 = {'a': {'1': {'c': {'d': {'e': 2}}}}, 'c': {'e': {}}}` – onesiumus Nov 06 '19 at 20:00
Of course, it is impossible to distinguish when to key into a list or key into a dictionary without introducing more syntax to the xquery logic. – onesiumus Nov 06 '19 at 20:07

score 2 · Answer 7 · answered Sep 06 '11 at 13:12

2

More work would have to be put into how the XPath-like selector would work. '/' is a valid dictionary key, so how would

foo={'/':{'/':'eggs'},'//':'ham'}

be handled?

foo.select("///")

would be ambiguous.

answered Sep 06 '11 at 13:12

unutbu

842,883
184
1,785
1,677

Yes, you would need a parser for that. But what I am asking is for a xpath _like_ method. "morefoo.morebar" is fine by me. – RickyA Sep 06 '11 at 13:24
3

@RickyA: `'.'` is also a value dictionary key. The same problem would exist. `foo.select('...')` would be ambiguous. – unutbu Sep 06 '11 at 13:44

score 2 · Answer 8 · edited May 23 '17 at 12:00

2

Another alternative (besides that suggested by jellybean) is this:

def querydict(d, q):
  keys = q.split('/')
  nd = d
  for k in keys:
    if k == '':
      continue
    if k in nd:
      nd = nd[k]
    else:
      return None
  return nd

foo = {
  'spam':'eggs',
  'morefoo': {
               'bar':'soap',
               'morebar': {'bacon' : 'foobar'}
              }
   }
print querydict(foo, "/morefoo/morebar")

edited May 23 '17 at 12:00

Community

1
1

answered Sep 06 '11 at 13:30

MarcoS

13,386
7
42
63

this should be the solution – bfmcneill Nov 23 '20 at 22:31

score 1 · Answer 9 · edited Apr 18 '13 at 14:42

1

Is there any reason for you to the query it the way like the XPath pattern? As the commenter to your question suggested, it just a dictionary, so you can access the elements in a nest manner. Also, considering that data is in the form of JSON, you can use simplejson module to load it and access the elements too.

There is this project JSONPATH, which is trying to help people do opposite of what you intend to do (given an XPATH, how to make it easily accessible via python objects), which seems more useful.

edited Apr 18 '13 at 14:42

SingleNegationElimination

151,563
33
264
304

answered Sep 06 '11 at 13:17

Senthil Kumaran

54,681
14
94
131

1

The reason is that I want to split the data and the query. I want to be flexible in the query part. If I access it the nested way the query is hardcoded in the program. – RickyA Sep 06 '11 at 13:28
@RickyA, in the other comment you say morefoo.morebar is fine. Did you check the JSONPATH project (Download and look at the source and tests). – Senthil Kumaran Sep 06 '11 at 13:30
I did take a look at JSONPATH, but my input is not text/json. It's nested dictionaries. – RickyA Sep 06 '11 at 13:44
@RickyA's question is super valuable when using mongodb, for example. If you want to iterate over nested keys in a BSON document, this is necessary. – Mittenchops Jul 02 '13 at 14:47

score 0 · Answer 10 · answered Sep 23 '18 at 23:32

def Dict(var, *arg, **kwarg):
  """ Return the value of an (imbricated) dictionnary, if all fields exist else return "" unless "default=new_value" specified as end argument
      Avoid TypeError: argument of type 'NoneType' is not iterable
      Ex: Dict(variable_dict, 'field1', 'field2', default = 0)
  """
  for key in arg:
    if isinstance(var, dict) and key and key in var:  var = var[key]
    else:  return kwarg['default'] if kwarg and 'default' in kwarg else ""   # Allow Dict(var, tvdbid).isdigit() for example
  return kwarg['default'] if var in (None, '', 'N/A', 'null') and kwarg and 'default' in kwarg else "" if var in (None, '', 'N/A', 'null') else var

foo = {
  'spam':'eggs',
  'morefoo': {
               'bar':'soap',
               'morebar': {'bacon' : 'foobar'}
              }
   }
print Dict(foo, 'morefoo', 'morebar')
print Dict(foo, 'morefoo', 'morebar', default=None)

Have a SaveDict(value, var, *arg) function that can even append to lists in dict...

score 0 · Answer 11 · answered Nov 09 '21 at 12:02

I reference form this link..

Following code is for json xpath base parse implemented in python :

import json
import xmltodict

# Parse the json string
class jsonprase(object):
    def __init__(self, json_value):
        try:
            self.json_value = json.loads(json_value)
        except Exception :
            raise ValueError('must be a json str value')


    def find_json_node_by_xpath(self, xpath):
        elem = self.json_value
        nodes = xpath.strip("/").split("/")
        for x in range(len(nodes)):
            try:
                elem = elem.get(nodes[x])
            except AttributeError:
                elem = [y.get(nodes[x]) for y in elem]
        return elem

    def datalength(self, xpath="/"):
        return len(self.find_json_node_by_xpath(xpath))

    @property
    def json_to_xml(self):
        try:
            root = {"root": self.json_value}
            xml = xmltodict.unparse(root, pretty=True)
        except ArithmeticError :
            pyapilog().error(e)
        return xml

Test Json :

{
    "responseHeader": {
        "zkConnected": true,
        "status": 0,
        "QTime": 2675,
        "params": {
            "q": "TxnInitTime:[2021-11-01T00:00:00Z TO 2021-11-30T23:59:59Z] AND Status:6",
            "stats": "on",
            "stats.facet": "CountryCode",
            "rows": "0",
            "wt": "json",
            "stats.field": "ItemPrice"
        }
    },
    "response": {
        "numFound": 15162439,
        "start": 0,
        "maxScore": 1.8660598,
        "docs": []
    }
}

Test Code to read the values from above input json.

numFound = jsonprase(ABOVE_INPUT_JSON).find_json_node_by_xpath('/response/numFound')
print(numFound)

XPath like query for nested python dictionaries

11 Answers11

dict > jmespath

dict > xml > xpath

Json Pointer

Linked

Related