Streaming with ActionController::Live not working in production

Question

I'm working with Ruby on Rails 4.1, streaming data from a Sidekiq process using ActionController::Live. In development, my streaming is working fantastic. In production (using Nginx/Puma), it's not going quite as well. Here's what's happening.

In production, referring to the image below of my Firebug, the "/events" is being fired multiple times. Why would my EventSource be fired repeatedly as opposed to waiting for my data? This does not happen in development.

Firebug EventSource

As long as my Sidekiq process is running, it will fire repeatedly at random intervals. As soon as my sidekiq process is complete, it will hang and not fire off any more. Then that last one will eventually time out (see red text in the image)

Here is my coffeescript:

source = new EventSource('/events')
source.addEventListener 'my_event', (e) ->
    console.log 'Got a message!'
    # Add the content to the screen somewhere

Referring to the Firebug image, it's almost like it times out during my Sidekiq process. I'm posting to redis from my worker.

Initializer

REDIS = Redis.new(url: 'redist://localhost:6379')

Sidekiq Worker

REDIS.publish('my_event', 'some data')

Then I have the controller action that the EventSource is hooked up to:

def events
  response.headers["Content-Type"] = "text/event-stream"
  redis = Redis.new(url: "redist://localhost:6379")
  # blocks the current thread
  redis.subscribe(['my_event', 'heartbeat']) do |on|
    on.message do |event, data|
      if event == 'heartbeat'
        response.stream.write("event: heartbeat\ndata: heartbeat\n\n")
      elsif event == 'my_event'
        response.stream.write("event: #{event}\n")
        response.stream.write("data: #{data.to_json}\n\n")
      end
    end
  end
rescue IOError
  logger.info 'Events stream closed'
ensure
  logger.info 'Stopping events streaming thread'
  redis.quit
  response.stream.close
end

Again, this works 100% fantastic in development. My data from Sidekiq streams perfectly to the browser. The heartbeat stuff above is because of this answer (I had the same issue)

Does anyone see something I'm doing wrong? Am I missing some form of setup for ActionController::Live to work in production with Nginx/Puma?

IMPORTANT NOTE

At the VERY END of each of those requests from the image (where it seems to time out and attempt to do the next one), I get one line of data successfully. So something is working it seems. But the single EventSource is not staying open and listening for the data long enough, it just keep repeating.

@Paul R, thanks for the tag correction. I didn't look close enough at that one! — ardavis, Jan 15 '15 at 14:39
No problem - it happens all the time - I get email notifications for the `sse` tag and about 1 in 10 questions are about Server Sent Events rather than Streaming SIMD Extensions. Acronym overload! — Paul R, Jan 15 '15 at 17:07

score 6 · Accepted Answer · edited May 23 '17 at 10:29

6

I had a similar issue with a recent project. I found it was my nginx config.

Add the following to the location section:

proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding off;
proxy_buffering off;
proxy_cache off;

This information was obtained from the following Stack Overflow discussion:

EventSource / Server-Sent Events through Nginx

Hope this helps.

edited May 23 '17 at 10:29

Community

1
1

answered Jan 15 '15 at 01:43

pvawser

98
1
7

Thanks! I'll try to give that a shot when I return to work tomorrow. – ardavis Jan 15 '15 at 13:12
That was it! I've been struggling with this for days, just a few simple lines fixes everything. Thank you for finding that. – ardavis Jan 16 '15 at 14:06
I don't know if it's related to this Nginx configuration, but it seems that my stream is successful until I take a break from sending data. It's almost like it times out. Then when I resume sending data, nothing shows up on the screen, even though I see the redis messages in the monitor. Thoughts? – ardavis Jan 20 '15 at 20:00
What rails server are you using? I read somewhere that when using Puma, if a new puma forks, the you may need to re-establish the redis connection. You do that in the puma config form memory. – pvawser Jan 20 '15 at 22:14
also important to note that thin can't be used, puma recommended – brauliobo Aug 01 '17 at 17:23

Streaming with ActionController::Live not working in production

1 Answers1