7

When I deploy a new version of my service to Heroku, what happens exactly?

Suppose I have N web dynos online right now, M of them currently servicing requests.

  • Do all of them shut down before the new version starts coming online? What happens to any pending requests currently being serviced?
  • Is there downtime? (let's assume I just have a stateless service without any migrations)
  • Is there a hook for doing custom migrations (e.g. migrate database tables)?
  • Can I bring up N servers running the new version, have them start servicing requests, and bring the previous N servers down only once they're not servicing any requests?
  • Does the answer depend on the stack/language? (Aspen/Bamboo/Cedar, Ruby/Node.js/Java/...)

I didn't any official documentation about this, just contrary posts (some saying hot migrations are not possible, while others say there is no downtime). Are there any official details about the deployment process and the questions above?

Community
  • 1
  • 1
ripper234
  • 222,824
  • 274
  • 634
  • 905

2 Answers2

17

Here is what happens during a Heroku deploy (current as of 10/20/2011*)[1]:

  • Heroku receives your git push
  • A new release is compiled from the latest version of your app and stored
  • [These happen roughly simultaneously]
    • The dyno grid is signalled to terminate[2] all running processes for your app
    • The dyno grid is signalled to start new processes for your app
    • The dyno grid is signalled to unidle all idle processes of your app
    • The HTTP router is signalled to begin routing HTTP traffic to the web dynos running the new release

The general takeaway is that to minimize any possible downtime you should minimize the boot time of your app.

By following careful migration practices, it is possible to push new code and then migrate while the app is running. Here's an example for Rails: http://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/

To minimize dropped connections during a restart, use a webserve that responds appropriately to SIGTERM by beginning a graceful shutdown (finish existing connections, dont take new ones). Newer versions of thin will handle SIGTERM correctly.

  1. This subject is the topic of much discussion internally and may change in the future.
  2. SIGTERM followed 10s later by SIGKILL if still running
David Dollar
  • 2,409
  • 17
  • 18
  • 2
    Thanks for the details. Are you a Heroku employee? Or if not, where did you find the details? Also, it would be interesting to know which of the supported web servers offers graceful shutdown like you described ... perhaps that is fit for another question. – ripper234 Oct 21 '11 at 09:23
0

I can answer "Is there a hook for doing custom migrations (e.g. migrate database tables)?" part of this question. I've handled running migrations by writing a shell script that does a "heroku rake db:migrate" immediately after I issue "git push heroku". I don't know if there is a more "hook" - y way to do that.

Wes Gamble
  • 777
  • 10
  • 29
  • If you use https://github.com/fastestforward/heroku_san it does something similar as part of the deployment process via rake deploy – John Beynon Oct 21 '11 at 08:10
  • You can also set up an HTTP deploy hook that calls an endpoint on the app itself which triggers migrations, etc. http://devcenter.heroku.com/articles/deploy-hooks#http_post_hook – David Dollar Nov 07 '11 at 15:41