3

According to https://developers.google.com/appengine/docs/python/config/cron cron jobs can run for 10 minutes. However, when I try and test it by going to the url for the cron job when signed in as an admin, it times out with a DeadlineExceededError. Best I can tell this happens about 30 seconds in, which is the non-cron limit for requests. Do I need to do something special to test it with the cron rules versus the normal limits?

Here's what I'm doing:

  • Going to the url for the cron job
  • This calls my handler which calls a single function in my py script
  • This function does a database call to google's cloud sql and loops through the resulting rows, calling a function on each row that use's ebay's api to get some data
  • The data from the ebay api call is stored in an array to all be written back to the database after all the calls are done.
  • Once the loop is done, it writes the data to the database and returns back to the handler
  • The handler prints a done message

It always has issues during the looping ebay api calls. It's something like 500 api calls that have to be made in the loop.

Any idea why I'm not getting the full 10 minutes for this?

Edit: I can post actual code if you think it would help, but I'm assuming it's a process that I'm doing wrong, rather than an error in the code since it works just fine if I limit the query to about 60 api calls.

doeiqts
  • 663
  • 1
  • 6
  • 14
  • You did not show your log. I expect you receive the deadline from an API call and not from the cron job. Probably your API deadline code or API deadline exception terminates your cron job. – voscausa Oct 22 '12 at 02:13

2 Answers2

10

The way GAE executes a cron job allows it to run for 10 min. This is probably done (i'm just guessing here) through checking the user-agent, IP address, or some other method. Just because you setup a cron job to hit a URL in your application doesn't mean a standard HTTP request from your browser will allow it to run for 10 minutes.

The way to test if the job works is to do so on the local dev server where there is no limit. Or wait until your cron job executes and check the logs for any errors.

Hope this helps!

someone1
  • 3,570
  • 2
  • 22
  • 35
  • 1
    Thanks, just uploaded it with a start time for in a few minutes and it ran the whole thing no problems. For some reason I thought it would distinguish between regular urls and cron urls when an admin made the request. – doeiqts Oct 22 '12 at 02:56
  • @doeiqts: could you elaborate on what "with a start time for in a few minutes" means? i'm having the same problem you had. – memius Jul 12 '13 at 22:11
  • @memius Sorry I never saw this before. I just meant that if the current time was 1:00p, I set the cron job to start at 1:05p so that I didn't have to wait very long for GAE to start running it. – doeiqts Apr 26 '14 at 22:25
2

Here is how you can clarify the exception and tell if it's a urlfetch problem. If the exception is:

* google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests;
* google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option;
* google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

then see https://developers.google.com/appengine/articles/deadlineexceedederrors as it's a urlfetch issue.

If it's the urlfetch that's timing out, try setting a longer duration (ex 60 sec.):

 result = urlfetch.fetch(url, deadline=60)
jamesmortensen
  • 33,636
  • 11
  • 99
  • 120
user1258245
  • 3,639
  • 2
  • 18
  • 23
  • 3
    Hi user, it seems like you're trying to seek clarification from the op. Answers shouldn't be used to seek clarification. Additionally, the cron job isn't treated as a cron if invoked directly from the url in the browser. – jamesmortensen Oct 22 '12 at 02:43
  • @jmort253 Sorry guy, I do tell what to do if it is the url fetch timing out. Not only that but I do tell how to confirm that it is the issue. You may be right about it not being treated as a cron job if accessed directly but saying that an answer which tells the op what to do, and how to confirm that is what is going wrong is not an answer is kinda out there. – user1258245 Oct 22 '12 at 02:54
  • 1
    No need to be so negative, @jmort253. Confusion between the different (identically-named) deadline exception was the first thing I thought of too; because the RPC deadline doesn't change. Glad it turned out the OP was hitting the cron URL from the browser. – Guido van Rossum Oct 22 '12 at 18:34
  • @GuidovanRossum and User, while this still isn't the problem, it could be helpful to future visitors. So I made an edit so it doesn't sound like a question for the op, and I reversed the downvote. Hope this helps! – jamesmortensen Oct 22 '12 at 21:05