How do you run multiple DelayedJob workers on a single Heroku dyno?

Question

I am having trouble getting my dynos to run multiple delayed job worker processes.

My Procfile looks like this:

worker: bundle exec script/delayed_job -n 3 start

and my delayed_job script is the default provided by the gem:

#!/usr/bin/env ruby

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize

When I try to run this either locally or on a Heroku dyno it exits silently and I can't tell what is going on.

foreman start
16:09:09 worker.1 | started with pid 75417
16:09:15 worker.1 | exited with code 0
16:09:15 system   | sending SIGTERM to all processes
SIGTERM received

Any help with either how to debug the issue or suggestions about other ways to go about running multiple workers on a single dyno it would be greatly appreciated.

score 15 · Accepted Answer · answered Jul 25 '14 at 04:40

You can use foreman to start multiple processes on the same dyno.

First, add foreman to your Gemfile.

Then add a worker line to your Procfile:

worker: bundle exec foreman start -f Procfile.workers

Create a new file called Procfile.workers which contains:

dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work

That will start 3 delayed_job workers on your worker dyno.

davogones · Answer 2 · 2016-12-30T19:17:28.797

8

Try changing your Procfile to:

worker: bundle exec script/delayed_job -n 3 run

Using start will create two daemons in the background and then immediately exit. Heroku thinks that your process crashed.

Using run keeps the workers in the foreground.

UPDATE: I use Delayed Job Worker Pool for this now.

edited Dec 30 '16 at 19:17

answered Feb 07 '15 at 00:13

davogones

7,321
31
36

Yes, been looking all over for this. This should be the accepted answer. – Vincil Bishop Dec 30 '16 at 04:03
1

Sorry to say this doesn't appear to work anymore. Run will now ignore the -n arguments and always run a single process. – DrewB Dec 22 '18 at 00:33

Jimmy Baker · Answer 3 · 2014-07-22T16:24:21.593

Short answer is that you can't do this with delayed_job. A dyno is a process and a single delayed_job worker works on a single process.

There are other solutions to this though. If you can switch over to using Sidekiq then you can run quite a few workers on a single process since Sidekiq workers use threading. The trade-off here is that your workers will need to be thread safe.

Check it out: http://sidekiq.org/

score 0 · Answer 4 · answered Jul 13 '20 at 09:44

I tried to use the accepted solution of @infused, but ran into an issue with the worker processes crashing on startup with Bundler::GemNotFound: Unable to find a spec satisfying web-console in the set. Perhaps the lockfile is corrupted?

The reason for that is that foreman start is looking for a .env file by default and read it: Forman Manual. Because the .env file was setup for a development environment, foreman was looking for the web-console gem and missing it. Conslusion: If you see your worker not starting, check if you have a .env file that is unsuited for production and rename it or specify another file on startup.

score 0 · Answer 5 · answered Feb 01 '22 at 03:22

0

I used the ruint buildpack to run a Procfile.worker in a single (worker) dyno: https://github.com/heroku/heroku-buildpack-runit

answered Feb 01 '22 at 03:22

Gabriel

964
9
8

How do you run multiple DelayedJob workers on a single Heroku dyno?

5 Answers5