Parse custom URIs with urlparse (Python)

Question

My application creates custom URIs (or URLs?) to identify objects and resolve them. The problem is that Python's urlparse module refuses to parse unknown URL schemes like it parses http.

If I do not adjust urlparse's uses_* lists I get this:

>>> urlparse.urlparse("qqqq://base/id#hint")
('qqqq', '', '//base/id#hint', '', '', '')
>>> urlparse.urlparse("http://base/id#hint")
('http', 'base', '/id', '', '', 'hint')

Here is what I do, and I wonder if there is a better way to do it:

import urlparse

SCHEME = "qqqq"

# One would hope that there was a better way to do this
urlparse.uses_netloc.append(SCHEME)
urlparse.uses_fragment.append(SCHEME)

Why is there no better way to do this?

urlparse also takes another param, I'm notin that it does not do any difference. (Example: `urlparse.urlparse("qqqq://base/id#hint", "http")` — u0b34a0f6ae, Sep 13 '09 at 15:40
I believe this question (or it's answers, depending how you look at it) [is out of date](http://stackoverflow.com/a/34902870/476716). — OrangeDog, Mar 18 '16 at 14:43

toothygoose · Answer 1 · 2011-07-08T07:58:22.190

22

You can also register a custom handler with urlparse:

import urlparse

def register_scheme(scheme):
    for method in filter(lambda s: s.startswith('uses_'), dir(urlparse)):
        getattr(urlparse, method).append(scheme)

register_scheme('moose')

This will append your url scheme to the lists:

uses_fragment
uses_netloc
uses_params
uses_query
uses_relative

The uri will then be treated as http-like and will correctly return the path, fragment, username/password etc.

urlparse.urlparse('moose://username:password@hostname:port/path?query=value#fragment')._asdict()
=> {'fragment': 'fragment', 'netloc': 'username:password@hostname:port', 'params': '', 'query': 'query=value', 'path': '/path', 'scheme': 'moose'}

edited Jul 08 '11 at 07:58

answered Jun 07 '11 at 11:00

toothygoose

1,292
1
10
9

But query still is not parsed properly... Thanks anyway. – Vladimir Mihailenco Jun 29 '11 at 10:30
using dir(urlparse) to find 5 variables seems indirect / fragile (what if urlparse changes in a new version, not to expose these internals?). thanks for the list though. – gatoatigrado Mar 28 '13 at 23:27

score 3 · Accepted Answer · answered Sep 13 '09 at 15:15

3

I think the problem is that URI's don't all have a common format after the scheme. For example, mailto: urls aren't structured the same as http: urls.

I would use the results of the first parse, then synthesize an http url and parse it again:

parts = urlparse.urlparse("qqqq://base/id#hint")
fake_url = "http:" + parts[2]
parts2 = urlparse.urlparse(fake_url)

answered Sep 13 '09 at 15:15

Ned Batchelder

364,293
75
561
662

I perfer my own workaround to this one; I would have to do this roundtrip all the time in my custom URL module. – u0b34a0f6ae Sep 13 '09 at 15:39
Fair enough: I didn't like relying on internals of the module, but I reasonable engineers can differ! – Ned Batchelder Sep 13 '09 at 18:23

score 3 · Answer 3 · answered Jun 04 '13 at 21:22

There is also library called furl which gives you result you want:

>>>import furl
>>>f=furl.furl("qqqq://base/id#hint");
>>>f.scheme
'qqqq' 

>>> f.host
'base'  
>>> f.path
Path('/id')
>>>  f.path.segments
['id']
>>> f.fragment                                                                                                                                                                                                                                                                 
Fragment('hint')   
>>> f.fragmentstr                                                                                                                                                                                                                                                              
'hint'

score 2 · Answer 4 · answered Jan 20 '16 at 14:33

2

The question appears to be out of date. Since at least Python 2.7 there are no issues.

Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
>>> import urlparse
>>> urlparse.urlparse("qqqq://base/id#hint")
ParseResult(scheme='qqqq', netloc='base', path='/id', params='', query='', fragment='hint')

answered Jan 20 '16 at 14:33

OrangeDog

36,653
12
122
207

Wanted to back this up further, as of 04/26/2016; also parses beyond the basics shown above: `weird_scheme = 'qqq://username:password@example.com/some/path?params=key#frag_ment'`. Then parse and show username: `urlparse(weird_scheme).username #'username'` or show query: `urlparse(weird_scheme).query) #'params=key'` – knickum Apr 26 '16 at 19:37

score 1 · Answer 5 · answered Feb 05 '11 at 13:33

Try removing the scheme entirely, and start with //netloc, i.e.:

>>> SCHEME="qqqq"
>>> url="qqqq://base/id#hint"[len(SCHEME)+1:]
>>> url
'//base/id#hint'
>>> urlparse.urlparse(url)
('', 'base', '/id', '', '', 'hint')

You won't have the scheme in the urlparse result, but you know the scheme anyway.

Also note that Python 2.6 seems to handle this url just fine (aside from the fragment):

$ python2.6 -c 'import urlparse; print urlparse.urlparse("qqqq://base/id#hint")'
ParseResult(scheme='qqqq', netloc='base', path='/id#hint', params='', query='', fragment='')

score 0 · Answer 6 · answered Dec 25 '13 at 15:35

You can use yurl library. Unlike purl or furl, it not try to fix urlparse bugs. It is new compatible with RFC 3986 implementation.

>>> import yurl
>>> yurl.URL('qqqq://base/id#hint')
URLBase(scheme='qqqq', userinfo=u'', host='base', port='', path='/id', query='', fragment='hint')

Parse custom URIs with urlparse (Python)

6 Answers6

Linked