I'm using python on google app engine, and keep getting google.appengine.api.urlfetch_errors.DeadlineExceededError on requests made from a machine which does some backend processing. The requests take approximately 60s, sometimes a little longer, so I've attempted to increase the deadline.
The requests are wrapped in a retry, and from the logs I can see that the time between retries is always ~60s. I assume this is either because I've configured things incorrectly, or misunderstand the limitations of the deadline.
The machine config is:
instance_class: B8
basic_scaling:
max_instances: 1
idle_timeout: 10m
The code I'm using is (redacted for simplicity):
from google.appengine.api import urlfetch
from retrying import retry
timeout = 600
retries = 10
@retry(
stop_max_attempt_number=retries,
wait_exponential_multiplier=1000,
wait_exponential_max=1000*60*5
)
def fetch(url):
"""Fetch remote data, retrying as necessary"""
urlfetch.set_default_fetch_deadline(timeout)
result = urlfetch.fetch(url)
if result.status_code != 200:
raise IOError("Did not receive OK response from server")
return result.content
data = fetch(config['url'])
I've tried setting the deadline explicitly as urlfetch.fetch(url, deadline=timeout) but setting the default seems to be the approach most people suggest.
Can anyone clarify whether there is a maximum value which can be set for deadline?
