Puma server hangs after capistrano "puma:restart" signal is sent

Question

When I deploy my application for the first time using cap [env] deploy, everything works as expected. The code is deployed and the puma server is started successfully with puma:start.

capistrano output of puma starting

 * 2015-01-09 12:19:37 executing `puma:start'
 * executing "cd /path/to/app/current && bundle exec puma -q -d -e production -C ./config/puma/production.rb"
   servers: ["example.com"]
   [example.com] executing command
** [out :: example.com] Puma starting in single mode...
** [out :: example.com] * Version 2.9.2 (ruby 2.1.5-p273), codename: Team High Five
** [out :: example.com] * Min threads: 0, max threads: 16
** [out :: example.com] * Environment: production
** [out :: example.com] * Listening on unix:///path/to/app/shared/sockets/puma.sock
** [out :: example.com] * Daemonizing...

If I make a code change and attempt to re-deploy, instead of calling puma:start, capistrano calls puma:restart, which acts as if successful, but puma wasn't actually restarted.

 * 2015-01-09 12:27:56 executing `puma:restart'
 * executing "cd /path/to/app/current && bundle exec pumactl -S /path/to/app/shared/sockets/puma.state restart"
   servers: ["example.com"]
   [example.com] executing command
** [out :: example.com] Command restart sent success

At this point, if I refresh the web page, I get a 504 Gateway Time-out error. It's very similar to this issue..

As the person suggested, if I added workers 1 to my puma configuration file, restart would be working, but start/stop would not.

At my current state (without workers), if I do cap [env] puma:stop, it does not stop puma. It also does NOT delete any of these files:

/path/to/app/shared/pids/puma.pid
/path/to/app/shared/sockets/puma.sock
/path/to/app/shared/sockets/puma.state

Important Note

In my Rails server, I am using ActionController::Live with Rails 4.1. I am connected to an event stream with Javascript / Redis. I noticed this in my nginx error log:

nginx error log

2015/01/09 12:29:32 [error] 8992#0: *371 upstream timed out (110: 
Connection timed out) while reading response header from upstream, 
client: [ip], server: example.com, request: "GET /build_configurations/refresh_workers HTTP/1.1", 
upstream: "http://unix:///path/to/app/shared/sockets/puma.sock/build_configurations
/refresh_workers", host: "example.com", referrer: "https://example.com/"

All in all, how can I successfully use cap [env] deploy to deploy updates to my code while having puma start and restart okay?

Update

I found this issue that talks about restarting puma with ActionController::Live, but no solution seems present.

score 1 · Accepted Answer · edited May 23 '17 at 10:28

Using the information I found in this other Stack Overflow answer, I was able to implement a heartbeat that fixes this situation it seems. I haven't fully understood all of the mechanics myself, but I started with an initializer to start the heartbeat:

config/initializers/redis.rb

REDIS = Redis.new(url: "redist://localhost:6379")

heartbeat_thread = Thread.new do
  while true
    REDIS.publish('heartbeat','thump')
    sleep 2.seconds
  end
end

at_exit do
  # not sure this is needed, but just in case
  heartbeat_thread.kill
  REDIS.quit
end

My controller has:

def build_status_events
  response.headers["Content-Type"] = "text/event-stream"
  redis = Redis.new(url: "redist://localhost:6379")

  # blocks the current thread
  redis.psubscribe(['my_event', 'heartbeat']) do |on|
    on.pmessage do |pattern, event, data|
      response.stream.write("event: #{event}\n")
      if event == 'heartbeat'
        response.stream.write("data: heartbeat\n\n")
      else
        response.stream.write("data: #{data.to_json}\n\n")
      end
    end
  end
rescue IOError
  logger.info 'Events stream closed'
ensure
  logger.info 'Stopping Events streaming thread'
  redis.quit
  response.stream.close
end

I believe what happens is that the heartbeat gets posted, and if there's an error, the ensure block will be called and the subscription is closed. Then puma reboots as expected. If anyone has a better solution, or more information, please comment or add another answer.

Puma server hangs after capistrano "puma:restart" signal is sent

1 Answers1