39

The GitHub guys recently released their background processing app which uses Redis: http://github.com/defunkt/resque http://github.com/blog/542-introducing-resque

I have it working locally, but I'm struggling to get it working in production. Has anyone got a:

  1. Capistrano recipe to deploy workers (control number of workers, restarting them, etc)
  2. Deployed workers to separate machine(s) from where the main app is running, what settings were needed here?
  3. gotten redis to survive a reboot on the server (I tried putting it in cron but no luck)
  4. how did you work resque-web (their excellent monitoring app) into your deploy?

Thanks!

P.S. I posted an issue on Github about this but no response yet. Hoping some SO gurus can help on this one as I'm not very experienced in deployments. Thank you!

Brian Armstrong
  • 19,707
  • 17
  • 115
  • 144

4 Answers4

29

I'm a little late to the party, but thought I'd post what worked for me. Essentially, I have god setup to monitor redis and resque. If they aren't running anymore, god starts them back up. Then, I have a rake task that gets run after a capistrano deploy that quits my resque workers. Once the workers are quit, god will start new workers up so that they're running the latest codebase.

Here is my full writeup of how I use resque in production:

http://thomasmango.com/2010/05/27/resque-in-production

Tom Mango
  • 558
  • 5
  • 9
  • Great share. Dropped the file in my config folder and replaced the resque version number found from `bundle show resque`. Thank you! – Matt Gaidica Jul 09 '12 at 07:48
9

I just figured this out last night, for Capistrano you should use san_juan, then I like the use of God to manage deployment of workers. As for surviving a reboot, I am not sure, but I reboot every 6 months so I am not too worried.

Although he suggest different ways of starting it, this is what worked easiest for me. (Within your deploy.rb)

require 'san_juan'
after "deploy:symlink", "god:app:reload"
after "deploy:symlink", "god:app:start"

To manage where it runs, on another server, etc, he covers that in the configuration section of the README.

I use Passenger on my slice, so it was relatively easy, I just needed to have a config.ru file like so:

require 'resque/server'

run Rack::URLMap.new \
  "/" => Resque::Server.new

For my VirtualHost file I have:

<VirtualHost *:80>
        ServerName resque.server.com
        DocumentRoot /var/www/server.com/current/resque/public

        <Location />
          AuthType Basic
          AuthName "Resque Workers"
          AuthUserFile /var/www/server.com/current/resque/.htpasswd
          Require valid-user
        </Location>
</VirtualHost>

Also, a quick note. Make sure you overide the resque:setup rake task, it will save you lots of time for spawning new workers with God.

I ran into a lot of trouble, so if you need any more help, just post a comment.

Jamie Schembri
  • 6,047
  • 4
  • 25
  • 37
Garrett
  • 7,830
  • 2
  • 41
  • 42
  • Awesome stuff Garrett, thanks for your help! You're right this was REALLY confusing. I added some more details below that helped for my setup. – Brian Armstrong Nov 18 '09 at 20:39
  • Garrett, still a little confused on the god part. Are you using these with san_juan? http://github.com/defunkt/resque/tree/master/examples/god/ How does god get the new code from the repository onto the worker machines when you deploy? My workers are loading the environment. – Brian Armstrong Nov 18 '09 at 21:08
  • Also, I didn't really get what they meant with the custom `resque:setup` task. Did you just make one like this that loads it with the default settings? http://www.pastie.org/private/bxpevk0g7tfntmvhn6gew I didn't really understand this. – Brian Armstrong Nov 18 '09 at 21:10
  • And within Capistrano I removed san_juan, it was very problematic, so I made my own: http://gist.github.com/238268 – Garrett Nov 18 '09 at 21:21
  • https://gist.github.com/ed94682f99b6b47f02f7 - The redis one is sort of make shift, I based it off of the resque one. – Garrett Nov 18 '09 at 23:05
  • No problem, I know what it's like to try to delve into new technology and want to punch your computer :) – Garrett Nov 19 '09 at 19:57
4

Garrett's answer really helped, just wanted to post a few more details. It took a lot of tinkering to get it right...

I'm using passenger also, but nginx instead of apache.

First, don't forget you need to install sinatra, this threw me for a while. sudo gem install sinatra

Then you need to make a directory for the thing to run, and it has to have a public and tmp folder. They can be empty but the problem is that git won't save an empty directory in the repo. The directory has to have at least one file in it, so I made some junk files as placeholders. This is a weird feature/bug in git.

I'm using the resque plugin, so I made the directory there (where the default config.ru is). It looks like Garrett made a new 'resque' directory in his rails_root. Either one should work. For me...

cd MY_RAILS_APP/vendor/plugins/resque/
mkdir public 
mkdir tmp
touch public/placeholder.txt
touch tmp/placeholder.txt

Then I edited MY_RAILS_APP/vendor/plugins/resque/config.ru so it looks like this:

#!/usr/bin/env ruby
require 'logger'

$LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/lib')
require 'resque/server'

use Rack::ShowExceptions

# Set the AUTH env variable to your basic auth password to protect Resque.
AUTH_PASSWORD = "ADD_SOME_PASSWORD_HERE"
if AUTH_PASSWORD
  Resque::Server.use Rack::Auth::Basic do |username, password|
    password == AUTH_PASSWORD
  end
end

run Resque::Server.new

Don't forget to change ADD_SOME_PASSWORD_HERE to the password you want to use to protect the app.

Finally, I'm using Nginx so here is what I added to my nginx.conf

server {
  listen   80;
  server_name  resque.seoaholic.com;
  root /home/admin/public_html/seoaholic/current/vendor/plugins/resque/public;
  passenger_enabled on;
}

And so it gets restarted on your deploys, probably something like this in your deploy.rb

run "touch #{current_path}/vendor/plugins/resque/tmp/restart.txt"

I'm not really sure if this is the best way, I've never setup rack/sinatra apps before. But it works.

This is just to get the monitoring app going. Next I need to figure out the god part.

Brian Armstrong
  • 19,707
  • 17
  • 115
  • 144
  • To track and create an empty folder for deployment on production, just add a new file `.gitkeep` in that directory and ignore that directory in `.gitignore` file. Then commit it. – millisami Jan 09 '12 at 05:50
  • You can now mount resque-web as a rack app on a subpath of your production app: http://railscasts.com/episodes/271-resque?view=asciicast – m33lky Jan 25 '12 at 18:45
1

Use these steps instead of making configuration with web server level and editing plugin:

#The steps need to be performed to use resque-web with in your application

#In routes.rb

ApplicationName::Application.routes.draw do
  resources :some_controller_name
  mount Resque::Server, :at=> "/resque"
end

#That's it now you can access it from within your application i.e
#http://localhost:3000/resque

#To be insured that that Resque::Server is loaded add its requirement condition in Gemfile

gem 'resque', :require=>"resque/server"

#To add  basic http authentication add resque_auth.rb file in initializers folder and add these lines for the security

Resque::Server.use(Rack::Auth::Basic) do |user, password|
  password == "secret"
end

#That's It !!!!! :)

#Thanks to Ryan from RailsCasts for this valuable information.
#http://railscasts.com/episodes/271-resque?autoplay=true

https://gist.github.com/1060167

Mugur 'Bud' Chirica
  • 4,246
  • 1
  • 31
  • 34
Gul
  • 1,757
  • 2
  • 15
  • 26