Run resque in background

Question

I have a working rails app with a resque queue system which works very well. However, I lack a good way of actually demonizing the resque workers.

I can start them just fine by going rake resque:work QUEUE="*" but I guess it's not the point that you should have your workers running in the foreground. For some reason nobody seems to adress this issue. On the official resque github page the claim you can do something like this:

PIDFILE=./resque.pid BACKGROUND=yes QUEUE="*" rake resque:work

well - it doesn't fork into the background here at least.

score 15 · Answer 1 · answered Jan 27 '12 at 17:12

A +1 for resque-pool - it really rocks. We use it in combination with God to make sure that it is always available.

# Resque
God.watch do |w|

  w.dir = RAILS_ROOT

  w.name = "resque-pool"
  w.interval = 30.seconds
  w.start = "cd #{RAILS_ROOT} && sudo -u www-data sh -c 'umask 002 && resque-pool -d -E #{RAILS_ENV}'"
  w.start_grace = 20.seconds
  w.pid_file = "#{RAILS_ROOT}/tmp/pids/resque-pool.pid"

  w.behavior(:clean_pid_file)

  # restart if memory gets too high
  #w.transition(:up, :restart) do |on|
  #  on.condition(:memory_usage) do |c|
  #    c.above = 350.megabytes
  #    c.times = 2
  #  end
  #end

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end

This then gives you a really elegant way to reload code in your workers without interrupting jobs - simply kill -2 your resque-pool(s) when you deploy. Idle workers will die immediately, busy workers will die when they finish their current jobs, and God will restart resque-pool with workers using your new code.

These are our Resque tasks for Capistrano:

namespace :resque do

  desc "Starts resque-pool daemon."
  task :start, :roles => :app, :only => { :jobs => true } do
    run "cd #{current_path};resque_pool -d -e #{rails_env} start"
  end

  desc "Sends INT to resque-pool daemon to close master, letting workers finish their jobs."
  task :stop, :roles => :app, :only => { :jobs => true } do
    pid = "#{current_path}/tmp/pids/resque-pool.pid"
    sudo "kill -2 `cat #{pid}`"
  end

  desc "Restart resque workers - actually uses resque.stop and lets God restart in due course."
  task :restart, :roles => :app, :only => { :jobs => true } do
    stop # let God restart.
  end

  desc "List all resque processes."
  task :ps, :roles => :app, :only => { :jobs => true } do
    run 'ps -ef f | grep -E "[r]esque-(pool|[0-9])"'
  end

  desc "List all resque pool processes."
  task :psm, :roles => :app, :only => { :jobs => true } do
    run 'ps -ef f | grep -E "[r]esque-pool"'
  end

end

You might need to reconnect any DB connections when resque-pool forks workers - check the docs.

Any gists or updates to this setup as of Late 2013-2014? Examples anywhere? — Joshua F. Rountree, Jan 15 '14 at 13:51
Getting this: `resque:stop' is only run for servers matching {:roles=>:app, :only=>{:jobs=>true}}, but no servers matched Any suggestions? — Joshua F. Rountree, Jan 15 '14 at 15:19

score 13 · Answer 2 · answered Jan 09 '12 at 20:36

13

I had the same problem and the following works for me.

PIDFILE=./resque.pid BACKGROUND=yes QUEUE="*" rake resque:work >>  worker1.log &

You can also redirect STDERR to the same log file.

answered Jan 09 '12 at 20:36

mbsheikh

2,501
5
23
33

score 6 · Answer 3 · edited Jul 10 '13 at 00:00

6

To demonize a process you can use nohup:

nohup cmd &

On resque's github there is a config for monit, that shows how to use nohup, it looks something like this:

nohup bundle exec rake resque:work QUEUE=queue_name PIDFILE=tmp/pids/resque_worker_QUEUE.pid & >> log/resque_worker_QUEUE.log 2>&1

edited Jul 10 '13 at 00:00

mahemoff

44,526
36
160
222

answered Nov 13 '11 at 22:58

sparrovv

6,894
2
29
34

can u pls explain what does this cmd does in detail. I see no output in log file instead see o/p in nohup.out file. – codemilan Jun 12 '15 at 12:12

score 4 · Answer 4 · answered Jan 22 '12 at 05:25

4

Another option you should look into is using the resque pool gem to manage your workers.

You can run resque pool in background by using this command:

resque-pool --daemon --environment production

answered Jan 22 '12 at 05:25

Jhony Fung

2,970
4
29
32

score 2 · Answer 5 · answered Feb 28 '12 at 19:34

2

The BACKGROUND environment variable was added to Resque 1.20; make sure you're not using 1.19 or lower.

answered Feb 28 '12 at 19:34

Ian Terrell

10,667
11
45
66

I had 1.17 version and BACKGROUND=yes didn't work for me. Thanks! – Ivan Linko Dec 17 '13 at 11:39

score 1 · Answer 6 · answered May 30 '14 at 08:37

I also faced this issue, I start worker in cap task, but I got issue

BACKGROUND causes worker always in starting mode.
nohup process is killed right after finish, we must wait a couple seconds. But unable to append more command after '&'

At last, I must create a shell, let it sleep 5s after nohup... call. My code

desc 'Start resque'
task :start, :roles => :app do
  run("cd #{current_path} ; echo \"nohup bundle exec rake resque:work QUEUE=* RAILS_ENV=#{rails_env} PIDFILE=tmp/pids/resque_worker_1.pid &\nnohup bundle exec rake resque:work QUEUE=* RAILS_ENV=#{rails_env} PIDFILE=tmp/pids/resque_worker_2.pid &\nsleep 5s\" > startworker.sh ")
  run("cd #{current_path} ; chmod +x startworker.sh")
  run("cd #{current_path} ; ./startworker.sh")
  run("cd #{current_path} ; rm startworker.sh")
end

I know this is a situation solution. but it works well in my project

score 1 · Answer 7 · answered Jan 24 '12 at 13:53

One good way is to use God to manage it. It launches a daemonized version of Resque and monitor it. Actually, you can choose between using Resque as a daemon and letting God daemonize Resque. I choose option 2.

A resque.god file example :

rails_env   = ENV['RAILS_ENV']  || "production"
rails_root  = ENV['RAILS_ROOT'] || "/path/to/my/app/current"
num_workers = rails_env == 'production' ? 5 : 2

num_workers.times do |num|
  God.watch do |w|
    w.dir      = "#{rails_root}"
    w.name     = "resque-#{num}"
    w.group    = 'resque'
    w.interval = 30.seconds
    w.env      = {"QUEUE"=>"critical,mailer,high,low", "RAILS_ENV"=>rails_env}
    w.start    = "bundle exec rake -f #{rails_root}/Rakefile resque:work"
    w.stop_signal = 'QUIT'
    w.stop_timeout = 20.seconds

    w.uid = 'myappuser'
    w.gid = 'myappuser'

    w.behavior(:clean_pid_file)

    # restart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 350.megabytes
        c.times = 2
        c.notify = {:contacts => ['maxime'], :priority => 9, :category => 'myapp'}
      end
    end

    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end

    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end

      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end

    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
        c.notify = {:contacts => ['maxime'], :priority => 1, :category => 'myapp'}
      end
    end
  end
end

score 0 · Answer 8 · answered Nov 24 '15 at 10:30

You can manage your workers with this script. Commands available:

rake resque:start_workers
rake resque:stop_workers
rake resque:restart_workers

There is also included resque-scheduler. Comment this lines to disable it:

pid = spawn(env_vars, 'bundle exec rake resque:scheduler', ops_s)
Process.detach(pid)

Run resque in background

8 Answers8