1

I have moved away from using the python requests library as it was a bit fiddly to get working on Google App Engine. Instead I'm using urllib2 which has better support. Unfortunately the POST request that previously worked with the requests library no longer works with urllib2.

With requests the code was as follows

values = { 'somekey' : 'somevalue'}
r = requests.post(some_url, data=values)

With urllib2, the code is as follows

values = { 'somekey' : 'somevalue'}
data = urllib.urlencode(values)
req = urllib2.Request(some_url, data)
response = urllib2.urlopen(req)

Unfortunately the latter raises the following error

HTTP Error 405: Method Not Allowed

The url I'm posting to has the following form:

some_url = 'http://ec2-11-111-111-1.compute-1.amazonaws.com'

I have read that there is an issue with trailing slashes with urllib2, however when I add the slash as follows 

some_url = 'http://ec2-11-111-111-1.compute-1.amazonaws.com/'

I still get the same error.

There is no redirection at the destination - so the request shouldn't be transformed to a GET. Even if it were, the url actually accepts GET and POST requests. The EC2 linux instance is running django and I can see that it successfully receives the request and makes use of it - so doesn't return a 405. For some reason urllib2 is picking up a 405 and throwing the exception.

Any ideas as to what may be going wrong?

Edit 1:

As per @philip-tzou 's good call, the following information might help

print req.get_method()

yields POST

print req.header_items()

yields []

Edit 2

Adding a user agent header (as @padraic-cunningham suggests) didn't solve it unfortunately. I added the same header shown in the urllib2 example

    user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
    headers = {'User-Agent': user_agent}
    data = urllib.urlencode(values)
    req = urllib2.Request(some_url, data, headers)

Edit 3

As @furas suggested, I've sent the request I'm making to requestb.in to double check what's being sent. It is indeed a POST request being made, with the following headers:

Connection: close
Total-Route-Time: 0
X-Cloud-Trace-Context: ed073df03ccd05657<removed>2a203/<removed>129639701404;o=5
Connect-Time: 1
Cf-Ipcountry: US
Cf-Ray: <removed>1d155ac-ORD
Cf-Visitor: {"scheme":"http"}
Content-Length: 852
Content-Type: application/x-www-form-urlencoded
Via: 1.1 vegur
X-Request-Id: 20092050-5df4-42f8-8fe0-<removed>
Accept-Encoding: gzip
Cf-Connecting-Ip: <removed>
Host: requestb.in
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppEngine-Google; (+http://code.google.com/appengine; appid: s~<removed>)

and now req.header_items() yields [('User-agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)')]

by way of comparison, the headers from the requests POST are

Cf-Connecting-Ip: <removed>
Accept-Encoding: gzip
Host: requestb.in
Cf-Ipcountry: US
Total-Route-Time: 0
Cf-Ray: 2f42a45185<removed>-ORD
Connect-Time: 1
Connection: close
Cf-Visitor: {"scheme":"http"}
Content-Type: application/x-www-form-urlencoded
Content-Length: 852
X-Cloud-Trace-Context: 8932f0f9b165d9a0f698<removed>/1740038835186<removed>;o=5
Accept: */*
X-Request-Id: c88a29e1-660e-4961-8112-<removed>
Via: 1.1 vegur
User-Agent: python-requests/2.11.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~<removed>)
Community
  • 1
  • 1
user714852
  • 2,054
  • 4
  • 30
  • 52
  • Have you tried adding headers? requests adds common headers to each request so that may be the issue. – Padraic Cunningham Oct 18 '16 at 20:58
  • What's the results when you print `req.get_method()` and `req.header_items()`? – Philip Tzou Oct 18 '16 at 21:00
  • you can use http://httpbin.org to compare `urllib2` request with `requests.post` – furas Oct 18 '16 at 21:03
  • In your urllib code, where do you specify that your request is a `POST`? Does `Request()` default to `POST` if `data` is present? – John Gordon Oct 18 '16 at 21:06
  • @JohnGordon yes - the docs https://docs.python.org/2/howto/urllib2.html show that if data is present it defaults to a POST request. – user714852 Oct 18 '16 at 21:10
  • you can compare the header send by `request.post` and your `urllib2.urlopen` and see what is missing. – Enix Oct 19 '16 at 06:31
  • @Enix I've updated the question to include the headers from the two different request methods. Can't see anything obvious there... – user714852 Oct 19 '16 at 07:54
  • does `requests` work fine when posting data to your GAE server? – Enix Oct 19 '16 at 08:11
  • To clarify, the POST request is sent from GAE to an EC2 linux instance (running django). when sent via requests it works, but when sent via urllib2 it throws an exception. – user714852 Oct 19 '16 at 08:45

0 Answers0