Add params to given URL in Python

Question

Suppose I was given a URL.
It might already have GET parameters (e.g. http://example.com/search?q=question) or it might not (e.g. http://example.com/).

And now I need to add some parameters to it like {'lang':'en','tag':'python'}. In the first case I'm going to have http://example.com/search?q=question&lang=en&tag=python and in the second — http://example.com/search?lang=en&tag=python.

Is there any standard way to do this?

score 230 · Accepted Answer · edited Aug 31 '17 at 16:38

230

There are a couple of quirks with the urllib and urlparse modules. Here's a working example:

try:
    import urlparse
    from urllib import urlencode
except: # For Python 3
    import urllib.parse as urlparse
    from urllib.parse import urlencode

url = "http://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url_parts = list(urlparse.urlparse(url))
query = dict(urlparse.parse_qsl(url_parts[4]))
query.update(params)

url_parts[4] = urlencode(query)

print(urlparse.urlunparse(url_parts))

ParseResult, the result of urlparse(), is read-only and we need to convert it to a list before we can attempt to modify its data.

edited Aug 31 '17 at 16:38

Acsor

1,011
2
13
26

answered Mar 24 '10 at 09:23

Łukasz

35,061
4
33
33

17

You probably want to use `urlparse.parse_qs` instead of `parse_qsl`. The latter returns a list whereas you want a dict. See http://docs.python.org/library/urlparse.html#urlparse.parse_qs. – Florian Brucker Jun 06 '12 at 09:01
12

@florian : At least in python 2.7 you then need to call `urlencode` as `urllib.urlencode(query, doseq=True)`. Otherwise, parameters that existed in the original url are not preserved correctly (because they are returned as tuples from @parse_qs@ – rluba Sep 11 '12 at 10:17
5

I've rewritten this to work in Python 3 as well. [Code here](https://gist.github.com/rokcarl/20b5bf8dd9b1998880b7). – duality_ Jan 22 '16 at 11:48
16

The results of `urlparse()` and `urlsplit()` are actually `namedtuple` instances. Thus you can assign them directly to a variable and use `url_parts = url_parts._replace(query = …)` to update it. – Feuermurmel Apr 05 '16 at 14:08
Yes, @Feuermurmel's comment is right: it is more readable as a dict than a list. `url_parts.query` vs `url_parts[4]`. As @Feuermurmel mentions, you would then need to use `_replace` as `urlparse` returns a `ParseResult`, not a dict, so you can't do `url_parts.query = ...`. – Yann Dìnendal Dec 12 '16 at 11:01
6

Caution - this implementation removes repeated query parameters that some RESTful services use. With a little modification this can be fixed. query = urlparse.parse_qsl(url_parts[4]) query += params.items() But then if you want to replace exiting query params using dict, takes a little more. – ombre42 Feb 28 '17 at 16:14
1

This implementation will fail with a URL that does not yet have a slash after the domain name, such has "http://hello.com". This can be fixed with `if url_parts[4]: url_parts[2] = url_parts[2] or '/'`. – nicbou Feb 16 '18 at 09:59

score 85 · Answer 2 · answered Apr 21 '18 at 16:40

85

Outsource it to the battle tested requests library.

This is how I will do it:

from requests.models import PreparedRequest
url = 'http://example.com/search?q=question'
params = {'lang':'en','tag':'python'}
req = PreparedRequest()
req.prepare_url(url, params)
print(req.url)

answered Apr 21 '18 at 16:40

Varun

4,054
6
31
54

it is from requests packages? – perymerdeka Apr 29 '21 at 04:23
@perymerdeka yes – Zhymabek Roman Jun 18 '21 at 06:20

score 70 · Answer 3 · edited Apr 13 '23 at 16:07

Why

I've been not satisfied with all the solutions on this page (come on, where is our favorite copy-paste thing?) so I wrote my own based on answers here. It tries to be complete and more Pythonic. I've added a handler for dict and bool values in arguments to be more consumer-side (JS) friendly, but they are yet optional, you can drop them.

How it works

Test 1: Adding new arguments, handling Arrays and Bool values:

url = 'http://stackoverflow.com/test'
new_params = {'answers': False, 'data': ['some','values']}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test?data=some&data=values&answers=false'

Test 2: Rewriting existing args, handling DICT values:

url = 'http://stackoverflow.com/test/?question=false'
new_params = {'question': {'__X__':'__Y__'}}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test/?question=%7B%22__X__%22%3A+%22__Y__%22%7D'

Talk is cheap. Show me the code.

Code itself. I've tried to describe it in details:

from json import dumps

try:
    from urllib import urlencode, unquote
    from urlparse import urlparse, parse_qsl, ParseResult
except ImportError:
    # Python 3 fallback
    from urllib.parse import (
        urlencode, unquote, urlparse, parse_qsl, ParseResult
    )


def add_url_params(url, params):
    """ Add GET params to provided URL being aware of existing.

    :param url: string of target URL
    :param params: dict containing requested params to be added
    :return: string with updated URL
    
    >> url = 'https://stackoverflow.com/test?answers=true'
    >> new_params = {'answers': False, 'data': ['some','values']}
    >> add_url_params(url, new_params)
    'https://stackoverflow.com/test?data=some&data=values&answers=false'
    """
    # Unquoting URL first so we don't lose existing args
    url = unquote(url)
    # Extracting url info
    parsed_url = urlparse(url)
    # Extracting URL arguments from parsed URL
    get_args = parsed_url.query
    # Converting URL arguments to dict
    parsed_get_args = dict(parse_qsl(get_args))
    # Merging URL arguments dict with new params
    parsed_get_args.update(params)

    # Bool and Dict values should be converted to json-friendly values
    # you may throw this part away if you don't like it :)
    parsed_get_args.update(
        {k: dumps(v) for k, v in parsed_get_args.items()
         if isinstance(v, (bool, dict))}
    )

    # Converting URL argument to proper query string
    encoded_get_args = urlencode(parsed_get_args, doseq=True)
    # Creating new parsed result object based on provided with new
    # URL arguments. Same thing happens inside urlparse.
    new_url = ParseResult(
        parsed_url.scheme, parsed_url.netloc, parsed_url.path,
        parsed_url.params, encoded_get_args, parsed_url.fragment
    ).geturl()

    return new_url

Please be aware that there may be some issues, if you'll find one please let me know and we will make this thing better

Perhaps add a try except with from urllib.parse to include Python 3 support? Thanks for the snippet, very useful! — MattV, Jul 29 '15 at 14:01
Unencodes encoded urls such as ``http://stackoverflow.com/with%2Fencoded?data=some&data=values&answe%2rs=false``. Also, use three chevrons ```>>>``` to help doctests pick up your doctests — pelson, Apr 08 '20 at 09:00
Why not change `parsed_get_args = dict(parse_qsl(get_args))` to `parsed_get_args = parse_qs(get_args)` — Mattwmaster58, Apr 23 '20 at 17:27

score 60 · Answer 4 · edited Jan 28 '20 at 21:36

60

You want to use URL encoding if the strings can have arbitrary data (for example, characters such as ampersands, slashes, etc. will need to be encoded).

Check out urllib.urlencode:

>>> import urllib
>>> urllib.urlencode({'lang':'en','tag':'python'})
'lang=en&tag=python'

In python3:

from urllib import parse
parse.urlencode({'lang':'en','tag':'python'})

edited Jan 28 '20 at 21:36

radtek

34,210
11
144
111

answered Mar 24 '10 at 09:15

Mike Mueller

2,072
15
16

12

In python 3, this has been moved to [urllib.parse.urlencode](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlencode) – shad0w_wa1k3r Nov 21 '19 at 13:41

score 29 · Answer 5 · answered Jul 16 '14 at 22:23

29

You can also use the furl module https://github.com/gruns/furl

>>> from furl import furl
>>> print furl('http://example.com/search?q=question').add({'lang':'en','tag':'python'}).url
http://example.com/search?q=question&lang=en&tag=python

answered Jul 16 '14 at 22:23

surfeurX

1,262
12
16

Christophe Roussy · Answer 6 · 2018-04-10T07:19:11.037

26

If you are using the requests lib:

import requests
...
params = {'tag': 'python'}
requests.get(url, params=params)

edited Apr 10 '18 at 07:19

answered May 15 '17 at 12:02

Christophe Roussy

16,299
4
85
85

1

@chefhose the question is ... relative to what ? You are not in a web page, there is no context to be relative to. – Christophe Roussy Dec 23 '19 at 10:05

score 17 · Answer 7 · edited May 23 '17 at 12:03

17

Based on this answer, one-liner for simple cases (Python 3 code):

from urllib.parse import urlparse, urlencode


url = "https://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url += ('&' if urlparse(url).query else '?') + urlencode(params)

or:

url += ('&', '?')[urlparse(url).query == ''] + urlencode(params)

edited May 23 '17 at 12:03

Community

1
1

answered Nov 25 '15 at 18:39

Mikhail Gerasimov

36,989
16
116
159

4

I know you mentioned "simple cases", but to clarify: it won't work properly if there is an `?` in the anchor (`#?stuff`). – Yann Dìnendal Dec 12 '16 at 10:57

score 14 · Answer 8 · answered Sep 17 '18 at 17:57

I find this more elegant than the two top answers:

from urllib.parse import urlencode, urlparse, parse_qs

def merge_url_query_params(url: str, additional_params: dict) -> str:
    url_components = urlparse(url)
    original_params = parse_qs(url_components.query)
    # Before Python 3.5 you could update original_params with 
    # additional_params, but here all the variables are immutable.
    merged_params = {**original_params, **additional_params}
    updated_query = urlencode(merged_params, doseq=True)
    # _replace() is how you can create a new NamedTuple with a changed field
    return url_components._replace(query=updated_query).geturl()

assert merge_url_query_params(
    'http://example.com/search?q=question',
    {'lang':'en','tag':'python'},
) == 'http://example.com/search?q=question&lang=en&tag=python'

The most important things I dislike in the top answers (they are nevertheless good):

Łukasz: having to remember the index at which the query is in the URL components
Sapphire64: the very verbose way of creating the updated ParseResult

What's bad about my response is the magically looking dict merge using unpacking, but I prefer that to updating an already existing dictionary because of my prejudice against mutability.

I was about to write an answer and found this. (y) – Kireeti K Sep 10 '20 at 08:45 — Kireeti K, Sep 10 '20 at 08:45

unwind · Answer 9 · 2010-03-24T09:17:01.327

11

Yes: use urllib.

From the examples in the documentation:

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
>>> print f.geturl() # Prints the final URL with parameters.
>>> print f.read() # Prints the contents

edited Mar 24 '10 at 09:17

answered Mar 24 '10 at 09:10

unwind

391,730
64
469
606

1

Can you please give some brief example? – z4y4ts Mar 24 '10 at 09:11
1

f.read() will show you the HTML page. To see the calling url, f.geturl() – ccheneson Mar 24 '10 at 09:20
9

-1 for using a HTTP request for parsing a URL (which is actually basic string manipulation). Plus the actual problem is not considered, because you need to know how the URL looks like to be able to append the query string correctly. – poke Mar 24 '10 at 10:11
Either the author edited question either this answer is not related to it. – simplylizz Feb 27 '13 at 17:20
for python 3 this is now: `urllib.request.urlopen` and `urllib.parse.urlencode` – smoquet Oct 21 '21 at 13:31

score 10 · Answer 10 · answered Sep 29 '20 at 10:17

10

python3, self explanatory I guess

from urllib.parse import urlparse, urlencode, parse_qsl

url = 'https://www.linkedin.com/jobs/search?keywords=engineer'

parsed = urlparse(url)
current_params = dict(parse_qsl(parsed.query))
new_params = {'location': 'United States'}
merged_params = urlencode({**current_params, **new_params})
parsed = parsed._replace(query=merged_params)

print(parsed.geturl())
# https://www.linkedin.com/jobs/search?keywords=engineer&location=United+States

answered Sep 29 '20 at 10:17

revy

3,945
7
40
85

Watch out! This approach is using an internal function (indicated by the "_" before the function name) : `_replace`. It is not recommended to do so, because behavior of these internal functions may change or they may be removed without warning. – Adrian Nov 05 '21 at 16:40
2

According to another stack overflow comment, admittedly from a few years ago, that's not true @GrazingScientist: https://stackoverflow.com/questions/21628852/changing-hostname-in-a-url#comment58437696_21629125 – yellow-saint May 07 '22 at 17:37
1

@yellow-saint: You are absolutly right, even with Python 3.9 `_replace` is still a valid public method. I did not know that. So, thanks for pointing this out. Still, this is not the usual case for methods starting with an underscore ("_"). – Adrian May 08 '22 at 20:11
1

Just to add, `_replace` is part of the publicly advertised API, as it's documented in the docs: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse (```One such method is _replace(). The _replace() method will return a new ParseResult object replacing specified fields with new values```) – Adam Parkin Jan 13 '23 at 21:36

score 7 · Answer 11 · answered Mar 02 '12 at 18:17

7

I liked Łukasz version, but since urllib and urllparse functions are somewhat awkward to use in this case, I think it's more straightforward to do something like this:

params = urllib.urlencode(params)

if urlparse.urlparse(url)[4]:
    print url + '&' + params
else:
    print url + '?' + params

answered Mar 02 '12 at 18:17

Facundo Olano

2,492
2
26
32

5

How about .query instead of [4] ? – Debby Mendez Mar 19 '15 at 17:10

score 5 · Answer 12 · answered Mar 24 '10 at 09:14

Use the various urlparse functions to tear apart the existing URL, urllib.urlencode() on the combined dictionary, then urlparse.urlunparse() to put it all back together again.

Or just take the result of urllib.urlencode() and concatenate it to the URL appropriately.

score 4 · Answer 13 · answered Oct 17 '12 at 14:51

Yet another answer:

def addGetParameters(url, newParams):
    (scheme, netloc, path, params, query, fragment) = urlparse.urlparse(url)
    queryList = urlparse.parse_qsl(query, keep_blank_values=True)
    for key in newParams:
        queryList.append((key, newParams[key]))
    return urlparse.urlunparse((scheme, netloc, path, params, urllib.urlencode(queryList), fragment))

Monty · Answer 14 · 2012-08-08T15:25:28.010

Here is how I implemented it.

import urllib

params = urllib.urlencode({'lang':'en','tag':'python'})
url = ''
if request.GET:
   url = request.url + '&' + params
else:
   url = request.url + '?' + params

Worked like a charm. However, I would have liked a more cleaner way to implement this.

Another way of implementing the above is put it in a method.

import urllib

def add_url_param(request, **params):
   new_url = ''
   _params = dict(**params)
   _params = urllib.urlencode(_params)

   if _params:
      if request.GET:
         new_url = request.url + '&' + _params
      else:
         new_url = request.url + '?' + _params
   else:
      new_url = request.url

   return new_ur

urllib has been split up in Python 3. The urllib.urlencode() function is now urllib.parse.urlencode(), — Nada Touil, Apr 06 '22 at 23:03

score 2 · Answer 15 · answered Jan 24 '11 at 02:16

In python 2.5

import cgi
import urllib
import urlparse

def add_url_param(url, **params):
    n=3
    parts = list(urlparse.urlsplit(url))
    d = dict(cgi.parse_qsl(parts[n])) # use cgi.parse_qs for list values
    d.update(params)
    parts[n]=urllib.urlencode(d)
    return urlparse.urlunsplit(parts)

url = "http://stackoverflow.com/search?q=question"
add_url_param(url, lang='en') == "http://stackoverflow.com/search?q=question&lang=en"

Add params to given URL in Python

15 Answers15

Why

How it works

Talk is cheap. Show me the code.

Linked

Related