Frequent ChunkedEncodingError with Google App Engine / Requests

Question

I frequently have a ChunkedEncodingError when requesting servers using Requests (Python) and Google App Engine.

I looked at the answer from IncompleteRead using httplib but the problem is that I don't believe my issue is related to the querying server : I often get this error with various endpoints I'm using, including Intercom and FullContact.

I would have suspected the issue was related to the server of one service if the issue was always raised from the same server (for example, FullContact), but it's not the case. I've also encounter this issue with other, non related, requests.

So I'm suspecting the problem is either my code or Google. But from my code "point of view", I don't know what would be wrong. Here's a snippet:

result = requests.post(
    "https://api.intercom.io/companies",
    json={'some': 'data', 'that': 'are', 'sent': 'ok'},
    headers={'Accept': 'application/json'},
    auth=("app_id", "app_key",)
)

As you can see, the request is quite standard, nothing fancy. It also fails with something as simple as:

r = requests.get(url, params=params, timeout=3)

Does anyone experiences those issues with Google App Engine? Is there something I can do to avoid that?

Prior to 2.10.0, requests did not support GAE in its entirety. You should upgrade and see if the problem persists. — Ian Stapleton Cordasco, May 09 '16 at 20:37
Ok I switched to the latest version and pushed the code. So far so good. I'll wait a few days and come back to you. You can already post your comment as an answer, I will already upvote it. — Cyril N., May 10 '16 at 12:49

score 6 · Answer 1 · answered May 12 '16 at 08:37

There is a patch that (seems) to work on GAE. The issue is located in the iter_content function of requests, that uses the subsequent urllib3 library.

The issue is that Google override this library for their own implementation, but with a few changes that yield a ChunkedEncodingError at the Requests level.

I tried this patch, and so far, so good. In details, you must replace the following line in your requests/models.py file :

for chunk in self.raw.stream(chunk_size, decode_content=True):
    yield chunk

by :

if isinstance(self.raw._original_response._method, int):
    while True:
        chunk = self.raw.read(chunk_size, decode_content=True)
        if not chunk:
            break
        yield chunk
else:
    for chunk in self.raw.stream(chunk_size, decode_content=True):
        yield chunk

And the problem will stop.

I submitted an issue to talk about it on the Requests repository, and we'll see how this will evolve.

Frequent ChunkedEncodingError with Google App Engine / Requests

1 Answers1