Making a POST call instead of GET using urllib2

Question

There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.

I'm trying to do a simple POST call to a service:

url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
                         'age'  : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content

I can see the server logs and it says that I'm doing GET calls, when I'm sending the data argument to urlopen.

The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).

@IgnacioVazquez-Abrams This doc says 405 is not appropriate for GET and HEAD requests...https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses — theQuestionMan, May 19 '21 at 00:26

score 49 · Answer 1 · edited Mar 25 '16 at 19:25

Do it in stages, and modify the object, like this:

# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
    connection = opener.open(request)
except urllib2.HTTPError,e:
    connection = e

# check. Substitute with appropriate HTTP code.
if connection.code == 200:
    data = connection.read()
else:
    # handle the error case. connection.read() will still contain data
    # if any was returned, but it probably won't be of any use

This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.

This is so horrible to read; it's so hard finding just the code. — Daniel F, May 15 '17 at 12:29
I think it's useful to see the verbose way, perhaps for not the majority of cases, but sometimes when something special should be tuned. — Mark, Jun 21 '17 at 19:19

score 48 · Accepted Answer · edited May 23 '17 at 12:09

48

This may have been answered before: Python URLLib / URLLib2 POST.

Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.

edited May 23 '17 at 12:09

Community

1
1

answered Jun 14 '11 at 19:13

Gregg

3,236
20
15

Yep! I was missing a trailing slash in the URL. I'm voting up and marking as acepted. Thanks!! – alfonso.kim Jun 14 '11 at 20:03

Rob Cowie · Answer 3 · 2011-06-14T19:18:10.823

11

Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.

url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age'  : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()

As suggested by @Michael Kent do consider requests, it's great.

EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.

edited Jun 14 '11 at 19:18

answered Jun 14 '11 at 19:11

Rob Cowie

22,259
6
62
56

Thank you @Rob, I tried your suggestion but it didn't worked either. It turned out that I was missing a trailing slash in the URL. – alfonso.kim Jun 14 '11 at 20:06
@Akim no problem. Good thinking @Gregg :) – Rob Cowie Jun 14 '11 at 20:07
You can also pass data to a GET request, there is no way for urllib2.urlopen to know what you want from this context, that is why. – kevr Feb 08 '18 at 00:45
Just tested this, it works great as is. Thanks, @RobCowie! – Nader Alexan Mar 12 '21 at 10:32

score 6 · Answer 4 · edited Jul 31 '19 at 15:10

6

The requests module may ease your pain.

url = 'http://myserver/post_service'
data = dict(name='joe', age='10')

r = requests.post(url, data=data, allow_redirects=True)
print r.content

edited Jul 31 '19 at 15:10

chrki

6,143
6
35
55

answered Jun 14 '11 at 19:01

Michael Kent

1,736
12
11

52

OP asked "using urllib2" – Oded Breiner Jul 06 '16 at 09:57
Although it's works but OP said want to use urllib2 instead of requests, which is a 3rd party library. Using a 3rd party library may be helpful but also make the codes not portable, which may be the OP's original concern. – Pang Mar 03 '17 at 05:49
2

Indeed, this is absolutely useless for my situation. – kevr Feb 08 '18 at 00:43
one word: `Poratbility` – Poojan Jul 22 '19 at 18:49
3

a very interesting choice for your one word – Joseph Sheedy Aug 23 '19 at 00:12

score 5 · Answer 5 · answered Jun 14 '11 at 19:17

it should be sending a POST if you provide a data parameter (like you are doing):

from the docs: "the HTTP request will be a POST instead of a GET when the data parameter is provided"

so.. add some debug output to see what's up from the client side.

you can modify your code to this and try again:

import urllib
import urllib2

url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
                         'age'  : '10'})
content = opener.open(url, data=data).read()

score 1 · Answer 6 · answered Jun 14 '11 at 19:12

1

Try this instead:

url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
                         'age'  : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content

answered Jun 14 '11 at 19:12

rupello

8,361
2
37
34

score -2 · Answer 7 · answered Apr 16 '15 at 04:25

-2

url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()

May be this can help

answered Apr 16 '15 at 04:25

Vijay Chander

35
2

1

This is for a GET request, not a POST – Caumons Oct 29 '15 at 17:11

Making a POST call instead of GET using urllib2

7 Answers7

Linked