Parsing dictionary-like URL parameters in Python

Question

I'm working on implementing server-side filtering to serve KendoUI's Grid component, using Python.

The problem I'm facing is that the AJAX call that it generates by default seems to be incompatible with both Flask's built-in URL parser and Python's urlparse module.

Here's a contrived sample of the type of query string I'm having trouble with: a=b&c=d&foo[bar]=baz&foo[baz]=qis&foo[qis]=bar

Here's the result I'm going for:

{
    'a': 'b',
    'c': 'd',
    'foo': {
        'bar': 'baz',
        'baz': 'qis',
        'qis': bar'
    }
}

Unfortunately, here's the request.args you get from this, if passed to a Flask endpoint:

{
    'a': 'b',
    'c': 'd',
    'foo[bar]': 'baz'
    'foo[baz]': 'qis'
    'foo[qis]': 'bar'
}

Worse yet, in practice, the structure can be several layers deep. A basic call where you're filtering the column foo to only rows where the value is equal to 'bar' will produce the following:

{
    'filter[logic]': 'and',
    'filter[filters][0][value]': 'bar',
    'filter[filters][0][field]': 'foo',
    'filter[filters][0][operator]': 'eq'
}

I checked the RFC, and it requires that the query string contain only "non-hierarchical" data. While I believe it's referring to the object the URI represents, there is no provision for this type of data structure in the specification that I can find.

I begin to write a function that would take a dictionary of params and return the nested construct they represented, but I soon realized that it was nuanced problem, and that surely someone out there has had this trouble before.

Is anyone aware of either a module that will parse these parameters in the way I'm wanting, or an elegant way to parse them that I've perhaps overlooked?

Possible duplicate of http://stackoverflow.com/questions/7940085/getting-the-array-of-get-params-in-python — Nick ODell, Jan 14 '13 at 17:04
Well, kendoUI's Grid tries to use a GET param by default, for one. You might be able to override that, but it also seems to break the RESTful paradigm as well - sending JSON would require a POST, and you're not updating anything, only getting a list of items. (I could be WAY off base, so feel free to correct me) — Lyndsy Simon, Jan 14 '13 at 17:05
@NickODell Yes, that appears to be the same question, and I hadn't found that one in my search. Still, the answer doesn't solve my problem, it only confirms that there is no standard way of doing this. — Lyndsy Simon, Jan 14 '13 at 17:08
KendoUI's Grid control is powered by a DataSource object. The DataSource allows configuration of how it does CRUD via AJAX through jQuery's `$.ajax` method. It looks like I can use JSON, then. Still, this seems like it would be something handy to have for Python, since apparently both PHP and RoR handle nested dictionaries passed in the query string. — Lyndsy Simon, Jan 14 '13 at 17:14

Nick ODell · Accepted Answer · 2013-01-14T18:39:23.760

9

I just wrote a little function to do this:

from collections import defaultdict
import re
params = {
    'a': 'b',
    'c': 'd',
    'foo[bar]': 'element1',
    'foo[baz]': 'element2',
    'foo[qis]': 'element3',
    'foo[borfarglan][bofgl]': 'element4',
    'foo[borfarglan][bafgl]': 'element5',
}

def split(string, brackets_on_first_result = False):
    matches = re.split("[\[\]]+", string)
    matches.remove('')
    return matches

def mr_parse(params):
    results = {}
    for key in params:
        if '[' in key:
            key_list = split(key)
            d = results
            for partial_key in key_list[:-1]:
                if partial_key not in d:
                    d[partial_key] = dict()
                d = d[partial_key]
            d[key_list[-1]] = params[key]
        else:
            results[key] = params[key]
    return results
print mr_parse(params)

This should work to any nest level.

edited Jan 14 '13 at 18:39

answered Jan 14 '13 at 17:55

Nick ODell

15,465
3
32
66

Nice, after closer look it should do the trick for most cases. There are still some cases, where the query should give lists instead of dicts, but +1 – Tadeck Jan 14 '13 at 18:46
In the end, I went with a pre-existing library instead of building my own - but since "do it manually" is the approach I ended up with, I'm accepting your answer. For reference, here's the library I used: https://bitbucket.org/k_bx/jquery-unparam/ – Lyndsy Simon Jan 15 '13 at 21:36
This is wonderful, worked like a charm for Datatables server-side stuff: https://www.datatables.net/manual/server-side. I wish people wouldn't write their ajax calls assuming people are gonna use PHP, it's 2015 fergodssake – migreva Apr 07 '15 at 20:20
1

This will not work for arrays in query strings, baffles me that Python does not have this built in. – jshbrntt Sep 15 '17 at 14:55
@JoshuaBarnett Give me an example, and I'll fix it to work! – Nick ODell Sep 15 '17 at 15:21

score 4 · Answer 2 · answered Jan 14 '13 at 18:35

4

Some time ago I found this project: https://github.com/bernii/querystring-parser

It is specifically aimed at doing what you wanted.

However, outside PHP world, GET (and POST) parameters behave differently. Like they usually are implemented using multi-value dictiomaries. So the better idea may be to fit that or find a way compatible with both worlds.

Alternatively, you can really use JSON-seralized data in the request body (POST) and just treat the accessed resource as controller (the resource that does something, in this case searching for something, after you pass some data to it).

answered Jan 14 '13 at 18:35

Tadeck

132,510
28
152
198

"However, outside PHP world, GET (and POST) parameters behave differently." - Yeah, like, according the the RFC! Unfortunately, I'm working within the bounds of the tools I have at my disposal. I finally found a way to do this, by using a 3rd party library. It's obscure, and I intend to adopt it and package it properly. – Lyndsy Simon Jan 15 '13 at 21:31
Does anyone understand why this isn't the standard behavior of `urllib.parse.parse_qs`?! – mecampbellsoup Mar 08 '22 at 00:15

score 0 · Answer 3 · answered Jan 14 '13 at 17:16

0

You could send body in GET requests too. If all you have to send is some hierarchal data, might just json.dumps(data) at client and json.loads(data) on the server.

You can refer to such practices in httplib's documentation here: http://docs.python.org/2/library/httplib.html#httplib.HTTPConnection.request

answered Jan 14 '13 at 17:16

meson10

1,934
14
21

The server is supposed to assume the body of a GET request is null, I believe. http://stackoverflow.com/questions/10298899/how-to-send-data-in-request-body-with-a-get-when-using-jquery-ajax – Lyndsy Simon Jan 15 '13 at 21:33

Parsing dictionary-like URL parameters in Python

3 Answers3

Linked