2

I want to use the requests module in Google App Engine Python Standard Runtime Environment.

Quote from official Google Cloud docs:

You can use third-party libraries that are pure Python code with no C extensions, by copying the library into your application directory. If the third-party library is already built-in, bundled with the runtime, you can use the library without copying it into your app.

Third party libraries must be implemented as pure Python code with no C extensions. If copied to your application directory, they count towards file quotas because the library is uploaded to App Engine along with your application code.

requests isn't bundled with GAE, so I added it into my application folder according to the instructions.

requests required a few other modules that don't come with GAE, so I added all of them to my application folder:

  • certifi
  • chardet
  • idna
  • urllib3

Another problem came up. My request goes to the Stack Exchange API, which has the https:// protocol. Here's the error:

SSLError: HTTPSConnectionPool(host='api.stackexchange.com', port=443): Max retries exceeded with url: /2.2/1?site=stackoverflow (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

The ssl module is built into the GAE Python runtime, so I put the following in app.yaml:

libraries:
- name: webapp2
  version: latest

- name: ssl
  version: latest

It didn't work. I got the same error as before. I copied the SSL module folder into my application directory and did import ssl in main.py, but now it throws an exception asking for yet another module to be installed:

File "/Users/williamqin/Projects/stackpromo/ssl/__init__.py", line 61, in <module>
import _ssl2          # if we can't import it, let the error propagate
ImportError: No module named _ssl2

I searched all over the web for the _ssl2 Python module, but I couldn't find it anywhere!

How do I properly use the requests module in Google App Engine?

Community
  • 1
  • 1
clickbait
  • 2,818
  • 1
  • 25
  • 61
  • 2
    Are you using the Flex environment with Python 3? Those libraries are built into the Standard env, but not the Flex env. So, you will need to add it to your project, as you said in your 2nd try. What does your directory tree look like? – GAEfan Jul 27 '18 at 01:07

1 Answers1

4

This was a pain to setup for python 2.7 in GAE standard. It involved using app engine's beta version of their version of the python ssl library and a few other odds and ends.

I'm sure you'll face some differences for python3. This is were the key bits for me to get it working:

requests 2.18.2

requests_toolbelt 0.7.0

in appengine_config.py do this:

from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine

# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()

in app.yaml have these:

env_variables:
  GAE_USE_SOCKETS_HTTPLIB : 'true'


libraries:
- name: ssl
  version: "2.7.11"
- name: pycrypto
  version: "2.6"

FUTHERMORE, this got it working for me in production, but not on my localhost. In addition to a local lib directory with all my third party libraries, I had to setup an additional directory called localhost_libs and do this in appengine_config.py as well:

vendor.add('lib')

if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
    vendor.add('localhost_libs')

where I have pycrypto

Also, for a long time everyone had to do this too (eventually something changed in dev_appserver that stopped this from working): "ImportError: No module named _ssl" with dev_appserver.py from Google App Engine

Alex
  • 5,141
  • 12
  • 26