How do you perform HTTP requests from App Engine to internal services?

Question

I have an App Engine Standard (1st Gen) Python app, set up with serverless VPC access. I use requests 2.3 (I have issues with later versions on GAE) to do HTTP requests.

When I try doing an HTTP request to an internal IP address (10.x), it refuses to connect. However, serverless VPC access is working properly: I tested connecting to the same IP address on the same port using a non-HTTP client library (e.g. redis) from App Engine, and this works fine. It's just HTTP requests that fail.

I suspect the URL Fetch service is trying to do the HTTP requests, and fails on internal addresses. If so, is there a way to use requests without triggering the URL Fetch service? Or do you have any other clue what might be going wrong?

This might help: https://stackoverflow.com/questions/9604799/can-python-requests-library-be-used-on-google-app-engine — new name, Apr 05 '20 at 22:01
As far as I can tell from those links, you can only force the use of sockets globally by setting `GAE_USE_SOCKETS_HTTPLIB`. However, I'm experiencing more problems than usual by doing so; is there a way to use sockets just for a few specific calls? — Remko, Apr 08 '20 at 19:29

score 1 · Accepted Answer · answered Apr 09 '20 at 20:25

1

I think I found the answer by looking at the SDK source code:

from python_std_lib import httplib

gets you an HTTPLib that uses sockets, without having to override the global httplib (causing sockets to be used everywhere).

answered Apr 09 '20 at 20:25

Remko

823
6
16

How do you perform HTTP requests from App Engine to internal services?

1 Answers1