Correct way to make a Python HTTPS request using requests module?

Question

I have to make an HTTPS request in Python, and I am using the requests module to try to make my life easier.

The request needs to have a header and 3 FORM parameters URL encoded. This is what I am doing:

header = {'Content-type': 'application/x-www-form-urlencoded', 'Authorization':'Basic ' + encoded_string, 'Connection': 'Keep-Alive', 'Host':'host.host.com'}

payload='grant_type=authorization_code&code=' + request.args['code'] + '&state=' + request.args['state'] + '&redirect_uri=http://xxx.xyz.com/request_listener'

url = 'https://serviceprovider.xxx.com/auth/j_oauth_resolve_access_code'

response = requests.post(url, data=payload, headers=header, verify=False)

When I try to return the content or text of the response, I get an empty string. However, when I print the actual response object, it says it is a <Response [200]>, but if this were actually a 200 OK then the server I am POSTing too should go to the redirect_uri I have specified and I would get a notification there.

This is not happening, and I am mystified as to why.

http://docs.python.org/3/library/urllib.request.html#module-urllib.request — Samie Bencherif, Dec 10 '13 at 04:10
Some things to try: use `curl` to construct a request by hand, to make sure that you know what the contents of a valid request should be. If that works, then you know that there is a problem with the way you are building or delivering the request in Python. Construct the `payload` as a dict rather than a string. Try building a prepared request so you can examine the request state before sending it to the server. — Tim Pierce, Dec 10 '13 at 04:21

score 20 · Accepted Answer · edited Feb 10 '19 at 06:06

Your code is fighting the Requests library: you're doing a lot of stuff yourself that Requests will do for you.

Firstly, don't form-encode your data yourself, let Requests do it by providing a dictionary to data, like @flyer's answer suggested.

When you do this, Requests will also correctly set the Content-Type header, so you don't have to. Also, please don't send a Connection header: Requests will manage it for you. The same applies to Host headers: sending a Host header can only cause problems.

Finally, don't set the Authorization header yourself, let Requests do it by providing it with your authentication credentials. The idiomatic Requests code would be:

payload = {
    'grant_type': 'authorization_code', 
    'code': request.args['code'],
    'state': request.args['state'],
    'redirect_uri': 'http://xxx.xyz.com/request_listener',
}

url = 'https://serviceprovider.xxx.com/auth/j_oauth_resolve_access_code'

response = requests.post(url, data=payload, verify=False)

If that doesn't work, then I would suspect your payload data is bad.

Furthermore, OP may think that the Host header is preserved on redirect which it is not. — Ian Stapleton Cordasco, Dec 10 '13 at 14:07
I had to include the client id in base64 in the header, so I could not remove it, but I removed everything else. I did change the payload data to a dict, but I am receiving a 200 OK even when the request is failing. I suspect the issue is not with the request in that case.. — digerati32, Dec 11 '13 at 02:55

score 8 · Answer 2 · edited Feb 10 '19 at 06:10

8

It seems that there are two mistakes.

The first one:
When you want to post data, the data format should be like this:

payload = {
    'grant_type': 'authorization_code', 
    'code': request.args['code'],
    'state': request.args['state'],
    'redirect_uri': 'http://xxx.xyz.com/request_listener',
}

The second:
Requests could verify SSL certificates for https requests automatically and it sets verify=True as default. You set verify=False and that means that you want to ignore the ssl verification. That maybe not what you want. Here is the doc

edited Feb 10 '19 at 06:10

sigjuice

28,661
12
68
93

answered Dec 10 '13 at 04:23

flyer

9,280
11
46
62

The server certificates have a signing issue, so that is why `verify` is set to False. Changing the payload to be a dict causes a 400 error: where it says the request was syntactically incorrect. – digerati32 Dec 10 '13 at 04:35
@digerati32, could you give the complete url you want to request? – flyer Dec 10 '13 at 04:37

Correct way to make a Python HTTPS request using requests module?

2 Answers2

Linked