6

I'm running a series of Rails/Sinatra apps behind nginx + unicorn, with zero-downtime deploys. I love this setup, but it takes a while for Unicorn to finish restarting, so I'd like to send some sort of notification when it finishes.

The only callbacks I can find in Unicorn docs are related to worker forking, but I don't think those will work for this.

Here's what I'm looking for from the bounty: the old unicorn master starts the new master, which then starts its workers, and then the old master stops its workers and lets the new master take over. I want to execute some ruby code when that handover completes.

Ideally I don't want to implement any complicated process monitoring in order to do this. If that's the only way, so be it. But I'm looking for easier options before going that route.

Adam Lassek
  • 35,156
  • 14
  • 91
  • 107
  • 2
    Isn't it possible to monkey patch this file? http://bogomips.org/unicorn.git/tree/lib/unicorn/launcher.rb – Denis de Bernardy Nov 22 '13 at 19:24
  • Would not [Github way](https://github.com/blog/517-unicorn) (scroll down to “Slow deploys” section) be suitable for you? – Aleksei Matiushkin Nov 25 '13 at 11:18
  • @mudasobwa that's what I'm doing. I don't think you understood the question. – Adam Lassek Nov 26 '13 at 06:40
  • @AdamLassek Since you already use that schema, I don’t understand what prevents you from running `unicorn && my_ruby_script` instead of pure `unicorn`. Probably I indeed didn’t understand the question, sorry. – Aleksei Matiushkin Nov 26 '13 at 11:14
  • @mudasobwa the unicorn command is not restarted directly, we simply send the USR2 signal to the process to tell it to boot a new master. I don't care when the new master _starts_, I care when it takes over. – Adam Lassek Nov 27 '13 at 00:32
  • @AdamLassek Excuse me for spamming you, but wouldn’t `&& my_ruby_script` part of the command I suggested be executed _right after_ master is finished? – Aleksei Matiushkin Nov 27 '13 at 03:13

2 Answers2

2

I've built this before, but it's not entirely simple.

The first step is to add an API that returns the git SHA of the current revision of code deployed. For example, you deploy AAAA. Now you deploy BBBB and that will be returned. For example, let's assume you added the api "/checks/version" that returns the SHA.

Here's a sample Rails controller to implement this API. It assumes capistrano REVISION file is present, and reads current release SHA into memory at app load time:

class ChecksController
  VERSION = File.read(File.join(Rails.root, 'REVISION')) rescue 'UNKNOWN'

  def version
    render(:text => VERSION)
  end
end

You can then poll the local unicorn for the SHA via your API and wait for it to change to the new release.

Here's an example using Capistrano, that compares the running app version SHA to the newly deployed app version SHA:

namespace :deploy do
  desc "Compare running app version to deployed app version"
  task :check_release_version, :roles => :app, :except => { :no_release => true } do
    timeout_at = Time.now + 60
    while( Time.now < timeout_at) do
      expected_version = capture("cat /data/server/current/REVISION")
      running_version = capture("curl -f http://localhost:8080/checks/version; exit 0")

      if expected_version.strip == running_version.strip
        puts "deploy:check_release_version: OK"
        break
      else
        puts "=[WARNING]==========================================================="
        puts "= Stale Code Version"
        puts "=[Expected]=========================================================="
        puts expected_version
        puts "=[Running]==========================================================="
        puts running_version
        puts "====================================================================="
        Kernel.sleep(10)
      end
    end
  end
end

You will want to tune the timeouts/retries on the polling to match your average app startup time. This example assumes a capistrano structure, with app in /data/server/current and a local unicorn on port 8080.

Winfield
  • 18,985
  • 3
  • 52
  • 65
  • There are a couple differences with my production env, but I think I can make this work. – Adam Lassek Nov 27 '13 at 23:18
  • This task should be something you can use deliberately or chain after your unicorn restart cap command. Happy to help work through any particulars of your environment. – Winfield Nov 28 '13 at 03:16
0

If you have full access to the box, you could script the Unicorn script to start another script which loops through checking for /proc/<unicorn-pid>/exe which will link to the running process.

See: Detect launching of programs on Linux platform

Update

Based on the changes to the question, I see two options - neither of which are great, but they're options nonetheless...

  1. You could have a cron job that runs a Ruby script every minute which monitors the PID directory mtime, then ensure that PID files exist (since this will tell you that a file has changed in the directory and the process is running) then executes additional code if both conditions are true. Again, this is ugly and is a cron that runs every minute, but it's minimal setup.

  2. I know you want to avoid complicated monitoring, but this is how I'd try it... I would use monit to monitor those processes, and when they restart, kick off a Ruby script which sleeps (to ensure start-up), then checks the status of the processes (perhaps using monit itself again). If this all returns properly, execute additional Ruby code.

Option #1 isn't clean, but as I write the monit option, I like it even better.

Community
  • 1
  • 1
CDub
  • 13,146
  • 4
  • 51
  • 68
  • 1
    It's more complicated than that; the new master starts early on, but I want to know when it finishes loading the workers and takes over as master. – Adam Lassek Nov 08 '13 at 19:48