0

I have a BigQuery query that takes perhaps a minute several BigQuery queries that each take around 10-30 seconds to run that I have been trying to execute from Google App Engine. At one or more places in the call stack, an HTTP request is being killed with a DeadlineExceededError. Sometimes the DeadlineExceededError (unsure which kind) is raised as is, and sometimes it is translated to an HTTPException.

Following leads found in different SO posts, I have taken various steps to avoid the timeout:

  • Run the query in a task that is added to a GAE TaskQueue, setting the task_age_limit to 10m. (1)
  • Pass a timeoutMs flag to getQueryResults (called on a job object in Google's Python API) using a value of 599 * 1000 ~ 10 minutes. (2)
  • Just before the call to getQueryResults, call urlfetch.set_default_fetch_deadline(60), every time, in an attempt to ensure that the setting is local to the thread that is making the call. (3, 4, 5)

A gist of the relevant part of a typical stack trace can be found here. In a typical task execution, there will be a number of failures and then finally, perhaps, a success.

This answer seems to be saying that a urlfetch call will not be allowed to exceed 60 seconds on GAE, in any context (including a task). I doubt the queries are exceeding the hard limit in my case, so I'm probably missing an important step. Has anyone run into a similar situation and figured out what was going on?

Community
  • 1
  • 1
Eric Walker
  • 7,063
  • 3
  • 35
  • 38
  • 2
    Have you considered using async queries, rather than waiting for the result. https://developers.google.com/bigquery/querying-data#asyncqueries – Tim Hoffman May 15 '14 at 00:18
  • 1
    Yes, because the you fir off the query and then check every so often for the result set. This means you can run very long queries. – Tim Hoffman May 15 '14 at 04:01
  • For anything expected to be long running on BQ, you probably should run an async job, then use a queue to check whether the result is ready. UrlFetch does have a hard limit of 60 seconds, regardless of being on a backend or not. – Nick May 16 '14 at 00:36

0 Answers0