Python - make a POST request using Python 3 urllib

Question

I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch

In order to simulate clicking the 'Search' button without filling out any of the form, which adds data to the page. I got the POST header information by clicking on the button while looking at the network tab in Chrome Developer Tools. The reason I'm posting this instead of just copying solutions from the other similar problems is that I believe I may have not gotten the correct header information.

Is it properly formatted and did I grab the right information? I've never made a POST request before.

This is what I've managed to piece together:

import urllib.parse
import urllib.request


data = urllib.parse.urlencode({'Host': 'search.cpsa.ca', 'Connection': 'keep-alive', 'Content-Length': 23796,
                                     'Origin': 'http://search.cpsa.ca', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
                                     'Cahce-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest',
                                     'X-MicrosoftAjax': 'Delta=true', 'Accept': '*/*',
                                     'Referer': 'http://search.cpsa.ca/PhysicianSearch',
                                     'Accept-Encoding': 'gzip, deflate',
                                     'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
                                     'Cookie': 'ASP.NET_SessionId=kcwsgio3dchqjmyjtwue402c; _ga=GA1.2.412607756.1459536682; _gat=1'})


url = "http://www.musi-cal.com/cgi-bin/query?%s"

data = data.encode('ascii')
with urllib.request.urlopen("http://search.cpsa.ca/PhysicianSearch", data) as f:
    print(f.read().decode('utf-8'))

This solution outputs the page's HTML, but not with any of the data I wanted to retrieve from the POST request.

The POST data should correspond to the name attributes and values of the input elements of the form that you are submitting. You can pick up the name attributes by inspecting the html of the form. You're using the request headers as POST data - that won't work. Also, consider using requests package (http://docs.python-requests.org/en/master/), it's friendlier than urllib2. — snakecharmerb, Apr 07 '16 at 18:26
How do I specify the form? Or do I just need key-value pairs specifying the data? — Daniel Paczuski Bak, Apr 07 '16 at 18:44
In chrome, look at the POST request in the network tab like you did earlier and go to the bottom of the headers tab - there you will see the names and values whether it's a POST request or a GET request with query parameters — snakecharmerb, Apr 07 '16 at 18:50
I did this, and a POST request is happening, but it's not actually grabbing the page. This is what requests.text gives me: 1|#||4|50|pageRedirect||%2fError.aspx%3faspxerrorpath%3d%2fPhysicianSearch| — Daniel Paczuski Bak, Apr 07 '16 at 20:18

score 134 · Accepted Answer · answered Apr 07 '16 at 19:11

134

This is how you do it.

from urllib import request, parse
data = parse.urlencode(<your data dict>).encode()
req =  request.Request(<your url>, data=data) # this will make the method "POST"
resp = request.urlopen(req)

answered Apr 07 '16 at 19:11

C Panda

3,297
2
11
11

3

What if I want to make a POST request with empty body? – Oleg Yablokov Dec 03 '21 at 13:34
2

request.Request(..., method='POST'). https://docs.python.org/3/library/urllib.request.html#urllib.request.Request.method – Anupam Ghosh Dec 22 '21 at 11:10
1

It's recommended practice to do `with request.urlopen(req) as resp:` and do whatever you need to do with `resp` in that block. – mVChr Mar 09 '22 at 15:49
Thank you Sir. You save my time. Your code working perfect on Python3 – Ganesan J Jul 28 '22 at 10:04
So you mean the mistake in the question was the attempt to pass the URL and data to `urllib.request.urlopen()` directly? Where does the documentation say that that's wrong? – root Nov 01 '22 at 14:26
@c panda, what can be a cause of 502 bad gateway error when I am using this piece of code when sending POST request to my API? – punky Jun 29 '23 at 07:26

score 36 · Answer 2 · answered Feb 26 '18 at 22:36

36

Thank you C Panda. You really made it easy for me to learn this module.

I released the dictionary that we pass does not encode for me. I had to do a minor change -

from urllib import request, parse
import json

# Data dict
data = { 'test1': 10, 'test2': 20 }

# Dict to Json
# Difference is { "test":10, "test2":20 }
data = json.dumps(data)

# Convert to String
data = str(data)

# Convert string to byte
data = data.encode('utf-8')

# Post Method is invoked if data != None
req =  request.Request(<your url>, data=data)

# Response
resp = request.urlopen(req)

answered Feb 26 '18 at 22:36

Centos Newbie

493
5
6

10

Should really be a comment – hd1 Apr 22 '18 at 01:05
7

if the service is strict in content type it accepts and it is json, then the following is also needed: `req.add_header('Content-Type', 'application/json')` https://stackoverflow.com/a/9746432/158328 – D_K Apr 08 '19 at 10:33
12

You can combine the 3 manipulations you're doing to `data` in a single command: `request.urlopen(url='your url', data=bytes(json.dumps(dict_obj), encoding='utf-8'))` – Chen A. Jun 10 '19 at 18:56

score 15 · Answer 3 · edited Nov 15 '21 at 15:23

15

Set method="POST" in request.Request().

Sending a POST request without a body:

from urllib import request

req = request.Request('https://postman-echo.com/post', method="POST")
r = request.urlopen(req)
content = r.read()
print(content)

Sending a POST request with json body:

from urllib import request
import json

req = request.Request('https://postman-echo.com/post', method="POST")
req.add_header('Content-Type', 'application/json')
data = {
    "hello": "world"
}
data = json.dumps(data)
data = data.encode()
r = request.urlopen(req, data=data)
content = r.read()
print(content)

edited Nov 15 '21 at 15:23

mime

302
1
6
21

answered Nov 16 '20 at 14:59

Jossef Harush Kadouri

32,361
10
130
129

My problem was that I was passing the `data` kwarg during the construction of `request.Request`, as in `request.Request(..., data=...)`, where instead data should be passed to `urllib.request.urlopen`. – bool3max Jan 13 '21 at 22:50
this question should be in the top – urek mazino Mar 29 '22 at 18:31

iphaaw · Answer 4 · 2018-11-12T17:01:41.403

13

The above code encoded the JSON string with some extra \" that caused me a lot of problems. This looks like a better way of doing it:

from urllib import request, parse

url = "http://www.example.com/page"

data = {'test1': 10, 'test2': 20}
data = parse.urlencode(data).encode()

req = request.Request(url, data=data)
response = request.urlopen(req)

print (response.read())

edited Nov 12 '18 at 17:01

answered Nov 12 '18 at 16:01

iphaaw

6,764
11
58
83

score 4 · Answer 5 · answered Apr 21 '20 at 14:49

It failed when I use urlencode. So I use the following code to make a POST call in Python3:

from urllib import request, parse

data = b'{"parameter1": "test1", "parameter2": "test2"}'
req = request.Request("http://www.musi-cal.com/cgi-bin/query?%s", data)
resp = request.urlopen(req).read().decode('utf-8')
print(resp)

Python - make a POST request using Python 3 urllib

5 Answers5

Linked