delayed_job: how to force processing of failed jobs

Question

I am maintaining a rails app, on which i'm running the delayed_job gem for sending emails.

I just noticed that all my delayed jobs've been failing the last few days due to a bug in the application. Now the bug is fixed and I want to process the jobs asap, but they are with already too many failed attempts, and the worker pulls them from the database with a huge delay.

I've tried by updating the delayed_jobs table and set the number of attempts to a smaller number and the run_at attribute to the current time, but still didn't help.

Can you tell me how can I force the worker to execute them ?

score 10 · Answer 1 · answered Jul 19 '12 at 09:40

10

You can start it manually, try

Delayed::Job.all.each { |j| j.invoke_job }

or

Delayed::Job.all.each { |j| j.payload_object.perform }

answered Jul 19 '12 at 09:40

Marcin Olichwirowicz

880
7
12

that sounds pretty reasonable, I'll try it now :) – Evgeniya Manolova Jul 19 '12 at 10:11
so, i've tried both methods, but still the jobs stayed unprocessed. i did it the from rails console, any idea what am I doing wrong ? – Evgeniya Manolova Jul 19 '12 at 10:47

Evgeniya Manolova · Answer 2 · 2012-07-27T13:48:30.727

3

Ok, so finally i got it!

The trick was really to update the run_at attribute to the current time, but the current for the app - which was 3hours behind the database.

When I set it to now() - interval '3 hours' all jobs were processed.

edit:

@rodzyn, I've tried your suggestion, but still couldn't get it work:

[20] pry(main)> Delayed::Job.all.size
  Delayed::Backend::ActiveRecord::Job Load (0.6ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" 
=> 1
[21] pry(main)> Delayed::Job.first.invoke_job
  Delayed::Backend::ActiveRecord::Job Load (0.5ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" LIMIT 1
  Order Load (0.4ms)  SELECT "orders".* FROM "orders" WHERE "orders"."id" = $1 LIMIT 1  [["id", "328"]]
  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1
=> nil
[22] pry(main)> Delayed::Job.all.size
  Delayed::Backend::ActiveRecord::Job Load (0.6ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" 
=> 1
[23] pry(main)>

edited Jul 27 '12 at 13:48

answered Jul 19 '12 at 10:07

Evgeniya Manolova

2,542
27
21

6

It's hard to tell what this answer is actually suggesting as the solution. For me, I just update the following in a loop which will cause the failed jobs to be re-processed and deleted once they are successful: `Delayed::Job.where("failed_at is not null").each do |dj| dj.run_at = Time.now; dj.last_error = nil; dj.failed_at = nil; dj.save! end` – steakchaser May 13 '14 at 18:42
I agree @steakchaser , this answer doesn't contain enough information about the solution. It was a long time ago though and I don't remember very well the situation. I've unchecked it, and will leave the question without an accepted answer. – Evgeniya Manolova May 20 '14 at 09:22

score 3 · Answer 3 · answered Nov 12 '13 at 17:30

None of the existing answers has this exactly right, so I add this here.

The magic is to convince delayed jobs that these jobs really aren't failed, so via the rails db console:

-> rails db
development# update delayed_jobs set run_at = now() - interval '3 hours', attempts = 0, failed_at = null;
UPDATE 30
development# \q

-> rake jobs:workoff  # a good one to use, because it will return immediately if no jobs are found

Either a "failed_at" or attempts field can keep a job from running.

score 1 · Answer 4 · answered Jul 19 '12 at 09:45

Now the bug is fixed and I want to process the jobs asap, but they are with already too many failed attempts, and the worker pulls them from the database with a huge delay.

That means, and since the jobs are not deleted from the table, there are still some more tries left for delayed job. The default behavior of delayed job is to read 5 jobs from the queue when finding an available job. One way without much of the code change is to have a setting in delayed config which would pick up more jobs from the queue and execute. You can configure this by setting Delayed::Worker.read_ahead.

# config/initializers/delayed_job_config.rb
Delayed::Worker.destroy_failed_jobs = false
Delayed::Worker.read_ahead = 10

Delayed::Worker.destroy_failed_jobs to avoid deleting of jobs after maximum attempts, which is also a configurable item.

Delayed job checks for available jobs every 5 seconds in the database and it would try a particular job after 5 + N ** 4 seconds. So, say if a particular job has failed 24 times already, then its turn would arrive after 331781 seconds, ie., roughly after 3 days if I am not wrong.

Thanks for the advice. I've already gone through that, but it doesn't solve my problem - my aim was to force the execution before the next scheduled execution. — Evgeniya Manolova, Jul 19 '12 at 10:11

delayed_job: how to force processing of failed jobs

4 Answers4

Linked