Should Exception catch DeadlineExceededError exceptions?

Question

I have the code like below:

try:
    response = urlfetch.Fetch(url, deadline=60, headers=headers, follow_redirects=False)
except Exception, error_message:
    logging.exception('Failed, exception happened - %s' % error_message)

but sometimes it fails with DeadlineExceededError? Should not this DeadlineExceededError be caught by my code?

The exception details:

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~myapp/1.375970700643773844/myapp.py", line 424, in get
    url, response = call_api(origin=origin, destination=destination, date=date_.strftime('%Y-%m-%d'))
  File "/base/data/home/apps/s~myapp/1.375970700643773844/myapp.py", line 288, in call_api
    response = urlfetch.Fetch(url, deadline=60, headers=headers, follow_redirects=False)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 270, in fetch
    return rpc.get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
    return self.__get_result_hook(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 375, in _get_fetch_result
    rpc.check_success()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 585, in check_success
    self.__rpc, err)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 196, in Call
    for key, function, srv, num_args in self.__content:
DeadlineExceededError

Why ? If you wait for 60 seconds on a urlfetch, you may have already consumed time and the DeadlineExceededError is raised elsewhere. — Tim Hoffman, May 20 '14 at 16:04
@TimHoffman, right. But in my case code stops to execute, when I believe it shouldn't since I handle the exception. — LA_, May 20 '14 at 16:07
But the exception can be raised outside of that try/except block if this is a front facing request. The google scheduler can decide you have been running long enough, and raise DEE. What does the stacktrace show in your log. — Tim Hoffman, May 20 '14 at 16:10
@TimHoffman, I've updated the question. The stacktrace points to this particular line. — LA_, May 20 '14 at 16:20
Note this statement in the docs "If the DeadlineExceededError is caught but a response is not produced quickly enough (you have less than a second), the request is aborted and a 500 internal server error is returned." Why don't you try it with a 58sec deadline on the request and see what happens. — Tim Hoffman, May 20 '14 at 16:29

score 3 · Answer 1 · answered Jul 17 '14 at 21:44

To answer your specific question, "Should not this DeadlineExceededError be caught by my code?":

The DeadlineExceededError class extends BaseException instead of Exception, so your except clause will never be called.

from google.appengine.runtime import DeadlineExceededError
my_error = DeadlineExceededError()
isinstance(my_error, Exception)  # False
isinstance(my_error, BaseException)  # True

rogcg · Answer 2 · 2014-05-20T17:49:32.653

First approach

Try excepting the DeadlineExceededError class by importing

from google.appengine.runtime import DeadlineExceededError and doing it like this:

try:
    response = urlfetch.Fetch(url, deadline=60, headers=headers, follow_redirects=False)
except DeadlineExceededError, error_message:
    logging.exception('Failed, exception happened - %s' % error_message)

Read more about it on the documentation.

Second Approach

I've faced this error before, and an approach I've followed was not to set which exception class I was trying to catch. I just called except.

try:
    response = urlfetch.Fetch(url, deadline=60, headers=headers, follow_redirects=False)
except:
    logging.exception('First failure')
    # try again your request as suggested by the documentation
    try:
        response = urlfetch.Fetch(url, deadline=60, headers=headers, follow_redirects=False)
    except:
        logging.exception('Second failure')
        return None

But I wasn't able to catch the error message, and instead of just logging an error message, I tried the request once again when the exception was raised, as the link I posted above suggests.

Some good links you should read

You should read this about Dealing with DeadlineExceedError.

There is also Backends, which allow you to avoid this request timer; with backends, there is no time limit for generating and returning a request.

Hope this helps you.

Should Exception catch DeadlineExceededError exceptions?

2 Answers2

First approach

Second Approach

Some good links you should read

Linked