24

I'm running unicorn and am trying to get zero downtime restarts working.

So far it is all awesome sauce, the master process forks and starts 4 new workers, then kills the old one, everyone is happy.

Our scripts send the following command to restart unicorn:

kill -s USR2 `cat /www/app/shared/pids/unicorn.pid`

On the surface everything looks great, but it turns out unicorn isn't reloading production.rb. (Each time we deploy we change the config.action_controller.asset_host value to a new CDN container endpoint with our pre-compiled assets in it).

After restarting unicorn in this way the asset host is still pointing to the old release. Doing a real restart (ie: stop the master process, then start unicorn again from scratch) picks up the new config changes.

preload_app is set to true in our unicorn configuration file.

Any thoughts?

Michael Shimmins
  • 19,961
  • 7
  • 57
  • 90
  • Does your unicorn.rb config try to kill the process on .oldbin ? See solution: http://jessewolgamott.com/blog/2012/01/02/the-one-where-unicorn-does-not-update/ – Jesse Wolgamott Feb 22 '12 at 02:11
  • It is definately killing hte old master - once the restart is complete and the workers are ready, `ps -ef | grep unicorn` shows only the master and 4 worker processes. The master's pid is different to the old master's pid before issuing the USR2. – Michael Shimmins Feb 23 '12 at 00:09

2 Answers2

24

My guess is that your unicorns are being restarted in the old production directory rather than the new production directory -- in other words, if your working directory in unicorn.rb is <capistrano_directory>/current, you need to make sure the symlink happens before you attempt to restart the unicorns.

This would explain why stopping and starting them manually works: you're doing that post-deploy, presumably, which causes them to start in the correct directory.

When in your deploy process are you restarting the unicorns? You should make sure the USR2 signal is being sent after the new release directory is symlinked as current.

If this doesn't help, please gist your unicorn.rb and deploy.rb; it'll make it a lot easier to debug this problem.

Veraticus
  • 15,944
  • 3
  • 41
  • 45
  • 5
    Indirectly answered the question - thanks! The restart was being issued after the symlink was complete, but you got me looking in the right place. The unicorn config had the working directory set to a `File.expand` statement that was resolving to the target of the symlink, not the symlink itself. I changed it to the absolute path of the symlink and it works perfectly. – Michael Shimmins Feb 26 '12 at 08:00
  • (PS - I'll award the bounty tomorrow when the clock has run down). – Michael Shimmins Feb 26 '12 at 08:00
  • I had this problem as well, turned out I was doing `cd #{release_path} &&` instead of `cd #{current_path} &&` when restarting unicorn. Thanks for helping me realize that. – Travis Mar 15 '12 at 17:39
  • Hi man: If I manually send USR2 signal from the shell but the new configuration is not picked up, what can be the cause? – Phương Nguyễn Dec 21 '13 at 00:25
  • @MichaelShimmins Can you post the actual change. I am a bit confused here. I have been trying for around a day now and still I am unable to make this work. – Anirudhan J Dec 25 '13 at 13:58
  • @AnirudhanJ - was a while ago (almost two years) but had a look through git history and it looks like this was the change: `- working_directory File.expand_path('../..', __FILE__) + working_directory '/www/lexim/current'` – Michael Shimmins Dec 29 '13 at 02:52
  • Thank you Michael. I tried various ways and ended up hard coding the absolute path for working_directory. I would love to know if there is any way to keep the working_directory dynamic. – Anirudhan J Dec 29 '13 at 07:12
7

Keep in mind that: your working directory in unicorn.rb should be : /your/cap/directory/current

NOT be: File.expand_path("../..", FILE)

Because the unicorn and linux soft link forking error: soft link can not work well.

for example:

cd current #current is a soft link to another directory

... ...

when we get our working directory, we got the absolute path not the path in "current"

fantaxy025025
  • 809
  • 8
  • 7