17

I'm using python-request on Google App Engine and it's not working as expected for HTTPS. Let's see an example:

import requests
requests.get('https://www.digitalocean.com')

That line works perfectly if I execute it in a terminal. Response is 200 OK (without redirects).

However, if I execute it on GAE a TooManyRedirects error is raised. Trying to figure out what's the problem I execute with allow_redirects=False and I can see that the response is a redirect (301) which points to the same url!!! ('location' header value is 'https://www.digitalocean.com'). This obviously (when allow_redirect=True) happens over and over again until the TooManyRedirects error is raised.

So it seems that python-requests is not working on GAE for HTTPS (I've tested with several URL). However HTTP works perfectly.

Any idea about what's happening?

Thanks in advance.

Curro
  • 1,331
  • 1
  • 13
  • 24

2 Answers2

17

Downgrading to requests==2.1.0 worked for me.

Having an up-to-date urllib3 is important for resolving an unrelated bug (import pwd, as I recall).

Hopefully App Engine fixes this soon, as requests won't.

EDIT:

I think you can also patch this in the latest requests by commenting lines 161-175 in sessions.py. Untested.

rattray
  • 5,174
  • 1
  • 33
  • 27
  • How do you downgrade to an earlier version of requests? – Connor Pearson Feb 07 '14 at 01:22
  • By now this seems to be the only way to make it works.... so it's a pretty valid response while we wait for the fix in GAE. Thanks rattray! good job! – Curro Feb 07 '14 at 09:24
  • Hi @rattay. The problem only seems to be to connect to Google IP addresses (e.g., Google oAuth), as they are blocked from the sockets API, as specified here: https://cloud.google.com/appengine/docs/python/sockets/ For that case, your patch does not seem to work anymore. In the latest version of requests (2.8.1), commenting our sessions.SessionRedirectMixin.rebuild_auth does not resolve the error: Failed to establish a new connection: [Errno 13] Permission denied – jacob Nov 24 '15 at 16:39
  • @jacob did you try downgrading to requests==2.1.0 ? – rattray Nov 24 '15 at 22:19
  • It works on 2.3 without any patches. I also posted this information on a requests git issue, and it looks like there is an official solution that is currently being worked out for the latest version 2.8+: https://github.com/kennethreitz/requests/issues/1905#issuecomment-159380704 – jacob Nov 25 '15 at 01:05
  • 1
    Here to confirm that 2.1.0 nor 2.3.0 work on the latest `gcloud 104 -- app-engine-python 1.9.35` – Josh Apr 08 '16 at 20:51
5

There is now a better solution than changing your requests version. As suggested in the official docs you can monkey patch requests to play nicely with Google App Engine.

First install requests-toolbelt:

pip install -t lib requests-toolbelt

Then in your main.py file (or equivalent):

import requests_toolbelt.adapters.appengine

requests_toolbelt.adapters.appengine.monkeypatch()
Alex
  • 18,484
  • 8
  • 60
  • 80