How to prevent python requests from percent encoding my URLs?

Question

I'm trying to GET an URL of the following format using requests.get() in python:

http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel

#!/usr/local/bin/python

import requests

print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)

However, the URL gets percent encoded and I don't get the expected response.

2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json

This works if I pass the URL directly:

url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)

Is there some way to pass the the parameters in their original form - without percent encoding?

Thanks!

Answer 1

It is not good solution but you can use directly string:

r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')

---

BTW:

Code which convert payload to this string

payload = {
    'format': 'json', 
    'key': 'site:dummy+type:example+group:wheel'
}

payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'

r = requests.get(url, params=payload_str)

---

EDIT (2020):

You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .

As I know requests also use urllib.parse.urlencode(...) for this but without safe=.

import requests
import urllib.parse

payload = {
    'format': 'json', 
    'key': 'site:dummy+type:example+group:wheel'
}

payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'

url = 'https://httpbin.org/get'

r = requests.get(url, params=payload_str)

print(r.text)

I used page https://httpbin.org/get to test it.

Answer 2

In case someone else comes across this in the future, you can subclass requests.Session, override the send method, and alter the raw url, to fix percent encodings and the like. Corrections to the below are welcome.

import requests, urllib

class NoQuotedCommasSession(requests.Session):
    def send(self, *a, **kw):
        # a[0] is prepared request
        a[0].url = a[0].url.replace(urllib.parse.quote(","), ",")
        return requests.Session.send(self, *a, **kw)

s = NoQuotedCommasSession()
s.get("http://somesite.com/an,url,with,commas,that,won't,be,encoded.")

Answer 3

The answers above didn't work for me.

I was trying to do a get request where the parameter contained a pipe, but python requests would also percent encode the pipe. So instead i used urlopen:

# python3
from urllib.request import urlopen

base_url = 'http://www.example.com/search?'
query = 'date_range=2017-01-01|2017-03-01'
url = base_url + query

response = urlopen(url)
data = response.read()
# response data valid

print(response.url)
# output: 'http://www.example.com/search?date_range=2017-01-01|2017-03-01'

Answer 4

The solution, as designed, is to pass the URL directly.

Answer 5

All above solutions don't seem to work anymore from requests version 2.26 on. The suggested solution from the GitHub repo seems to be using a work around with a PreparedRequest.

The following worked for me. Make sure the URL is resolvable, so don't use 'this-is-not-a-domain.com'.

import requests

base_url = 'https://www.example.com/search'
query = '?format=json&key=site:dummy+type:example+group:wheel'

s = requests.Session()
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)

Source: https://github.com/psf/requests/issues/5964#issuecomment-949013046

Answer 6

Please have a look at the 1st option in this github link. You can ignore the urlibpart which means prep.url = url instead of prep.url = url + qry

Answer 7

Requests and Urllib3 don't allow raw URLs anymore!

• Since those libraries enforce RFC3986 url encoding implicitly.
• Previously available workarounds are now defunct:
  • Even the prepped request workaround doesn't work anymore!

=> To perform raw path requests another library must be used like "http".

Raw Url Request Wrapper

• Wrapping the http library provides similar request methods but will not implicitly modify urls, thus allows to send urls as is
• Code:

import http.client
from urllib.parse import urlparse

class HttpWrapper:

    def parse_url(self, url):
        p_url = urlparse(url)
        host = p_url.hostname
        port = p_url.port
        path = p_url.path
        assert host
        assert port
        
        return host, port, path

    def request(self, url, method, headers={}, data=None):
        # Parse url
        host, port, path = self.parse_url(url)

        # Perform request
        try:
            conn = http.client.HTTPConnection(host, port)
            conn.request(method, path, data, headers)
            response = conn.getresponse()
            return response
        finally:
            conn.close()
    
    def get(self, url, headers):
        return self.request(url, "GET", headers)

    def post(self, url, headers, data):
        return self.request(url, "POST", headers, data)

    def put(self, url, headers, data):
        return self.request(url, "PUT", headers, data)
    
    def delete(self, url, headers):
        return self.request(url, "DELETE", headers)