Parsing values in params argument to request.get() which contain periods

Question

I am using the python package requests to make a get request to a server. My goal is to replicate certain known http requests and incorporated them into my python script. The original query string contains one key-value pair 'version=0%2E12' which I know is the url encoded equivalent of 'version=0.12'. Below is an example of what I am doing to replicate the http request.

params = {
          'version'='0.12',
          'language'='en',
          ...
          }
 resp = requests.get(url, params)

The resulting url query string has 'version=0.12' in it. If I try to change the setting of params to the following,

    params = {
          'version'='0%2E12',
          'language'='en',
          ...
          }

but the resulting query string contains 'version=0%252E31'. Essentially, requests is parsing the % to be url encoded.

How can I get requests to parse my params arguments with periods properly without manually writing the entire url?

`.` is a perfectly valid character in a URL-encoded value. Why do you *have* to have it encoded? — Martijn Pieters, Nov 30 '12 at 21:08
The original query string contains the %2E character and I want to reduce any errors that could crop up from making my requests different. — JonathanV, Nov 30 '12 at 21:10
Yes. The server sends back a 200 response code but the return body contains errors. — JonathanV, Nov 30 '12 at 21:12

Martijn Pieters · Accepted Answer · 2020-03-12T21:20:08.853

2

Note that . is a perfectly valid character in a URL-encoded value.

You'd need to encode the values yourself, bypassing the standard encoding. I'd use urllib.urlencode(), then replace the . characters by %2E by hand.

You then have to append the encoded string as a query parameter to the URL, and configure requests to not re-encode the values:

params = {
      'version'='0.12',
      'language'='en',
      ...
      }
params = urllib.urlencode(params)
params = params.replace('.', '%2E')
resp = requests.get('?'.join(url, params), config={'encode_uri': False})

The encode_uri parameter set to False does the trick here; it's this requests default that otherwise will re-encode the whole URL to ensure RFC compliancy.

edited Mar 12 '20 at 21:20

answered Nov 30 '12 at 21:16

Martijn Pieters

1,048,767
296
4,058
3,343

When I try to implement your solution I have an interesting result. Just before the request.get() call params has the correct '%2E' but once I watch the request go through in Fiddler2 I see that it is changed back to '.'. Do you know if having a period in the query string is an issue? – JonathanV Nov 30 '12 at 21:35
Maybe, however the original http request (also viewed in Fiddler2) has the %2E encoding. – JonathanV Nov 30 '12 at 21:53
1

@JonathanV: Found it; I ran through the code with a debugger, and `requests` is indeed re-encoding the full URL. There is a switch for that though, added to the answer. – Martijn Pieters Dec 01 '12 at 08:57
this doesnt work with version 1.04 of requests module i get request() got an unexpected keyword argument 'config' any ideas on how to get around this? – binhex Jan 04 '13 at 13:24
@user1582112: Requests 1.0x is heavily refactored. It may well have broken this approach, yes. It *could* be that it doesn't re-encode at all now. – Martijn Pieters Jan 04 '13 at 13:25

Parsing values in params argument to request.get() which contain periods

1 Answers1

Linked