Ruby on Rails 3: Streaming data through Rails to client

Question

I am working on a Ruby on Rails app that communicates with RackSpace cloudfiles (similar to Amazon S3 but lacking some features).

Due to the lack of the availability of per-object access permissions and query string authentication, downloads to users have to be mediated through an application.

In Rails 2.3, it looks like you can dynamically build a response as follows:

# Streams about 180 MB of generated data to the browser.
render :text => proc { |response, output|
  10_000_000.times do |i|
    output.write("This is line #{i}\n")
  end
}

(from http://api.rubyonrails.org/classes/ActionController/Base.html#M000464)

Instead of 10_000_000.times... I could dump my cloudfiles stream generation code in there.

Trouble is, this is the output I get when I attempt to use this technique in Rails 3.

#<Proc:0x000000010989a6e8@/Users/jderiksen/lt/lt-uber/site/app/controllers/prospect_uploads_controller.rb:75>

Looks like maybe the proc object's call method is not being called? Any other ideas?

score 70 · Answer 1 · edited Mar 19 '16 at 01:06

70

Assign to response_body an object that responds to #each:

class Streamer
  def each
    10_000_000.times do |i|
      yield "This is line #{i}\n"
    end
  end
end

self.response_body = Streamer.new

If you are using 1.9.x or the Backports gem, you can write this more compactly using Enumerator.new:

self.response_body = Enumerator.new do |y|
  10_000_000.times do |i|
    y << "This is line #{i}\n"
  end
end

Note that when and if the data is flushed depends on the Rack handler and underlying server being used. I have confirmed that Mongrel, for instance, will stream the data, but other users have reported that WEBrick, for instance, buffers it until the response is closed. There is no way to force the response to flush.

In Rails 3.0.x, there are several additional gotchas:

In development mode, doing things such as accessing model classes from within the enumeration can be problematic due to bad interactions with class reloading. This is an open bug in Rails 3.0.x.
A bug in the interaction between Rack and Rails causes #each to be called twice for each request. This is another open bug. You can work around it with the following monkey patch:
```
class Rack::Response
  def close
    @body.close if @body.respond_to?(:close)
  end
end
```

Both problems are fixed in Rails 3.1, where HTTP streaming is a marquee feature.

Note that the other common suggestion, self.response_body = proc {|response, output| ...}, does work in Rails 3.0.x, but has been deprecated (and will no longer actually stream the data) in 3.1. Assigning an object that responds to #each works in all Rails 3 versions.

edited Mar 19 '16 at 01:06

typeoneerror

55,990
32
132
223

answered Dec 01 '10 at 01:17

John

29,546
11
78
79

1

invaluable response, thank you. Used it to implement streaming templates for csv file: https://github.com/fawce/csv_builder – fawce Aug 11 '11 at 15:01
Thanks so much. Why are these methods deprecated and there is no official way of streaming data?! – m33lky Mar 11 '12 at 05:13
1

unfortunately this solution is not working for me. I started a new discussion here [link](http://stackoverflow.com/questions/14356704/rails-3-2-streaming-data) – dc10 Jan 16 '13 at 10:57
John, we run into memory issue with the code above. If we stream large amount of the data, it seems that it will consume large RAM and never release. We are running under Passenger 3.0.19. Do you have this issue? – DrChanimal May 10 '13 at 10:41
From your description it sounds like Passenger may be buffering the response in memory rather than streaming it to the client. I haven't used it so I can't say whether that's expected behavior or not. – John May 11 '13 at 02:47
Works with rails 4.0.0 too! – gucki Aug 21 '13 at 13:09
1

If it still not work (Rails 3.1.x), try to add the "Last-Modified" header (see response from Exequiel) – joel1di1 Sep 11 '13 at 12:37

paneer_tikka · Answer 2 · 2012-07-08T21:34:04.733

Thanks to all the posts above, here is fully working code to stream large CSVs. This code:

Does not require any additional gems.
Uses Model.find_each() so as to not bloat memory with all matching objects.
Has been tested on rails 3.2.5, ruby 1.9.3 and heroku using unicorn, with single dyno.
Adds a GC.start at every 500 rows, so as not to blow the heroku dyno's allowed memory.
You may need to adjust the GC.start depending on your Model's memory footprint. I have successfully used this to stream 105K models into a csv of 9.7MB without any problems.

Controller Method:

def csv_export
  respond_to do |format|
    format.csv {
      @filename = "responses-#{Date.today.to_s(:db)}.csv"
      self.response.headers["Content-Type"] ||= 'text/csv'
      self.response.headers["Content-Disposition"] = "attachment; filename=#{@filename}"
      self.response.headers['Last-Modified'] = Time.now.ctime.to_s

      self.response_body = Enumerator.new do |y|
        i = 0
        Model.find_each do |m|
          if i == 0
            y << Model.csv_header.to_csv
          end
          y << sr.csv_array.to_csv
          i = i+1
          GC.start if i%500==0
        end
      end
    }
  end
end

config/unicorn.rb

# Set to 3 instead of 4 as per http://michaelvanrooijen.com/articles/2011/06/01-more-concurrency-on-a-single-heroku-dyno-with-the-new-celadon-cedar-stack/
worker_processes 3

# Change timeout to 120s to allow downloading of large streamed CSVs on slow networks
timeout 120

#Enable streaming
port = ENV["PORT"].to_i
listen port, :tcp_nopush => false

Model.rb

  def self.csv_header
    ["ID", "Route", "username"]
  end

  def csv_array
    [id, route, username]
  end

Steven Yelton · Accepted Answer · 2010-10-04T16:06:40.817

16

It looks like this isn't available in Rails 3

https://rails.lighthouseapp.com/projects/8994/tickets/2546-render-text-proc

This appeared to work for me in my controller:

self.response_body =  proc{ |response, output|
  output.write "Hello world"
}

edited Oct 04 '10 at 16:06

answered Oct 04 '10 at 16:01

Steven Yelton

241
2
6

score 9 · Answer 4 · answered Apr 20 '12 at 20:02

9

In case you are assigning to response_body an object that responds to #each method and it's buffering until the response is closed, try in in action controller:

self.response.headers['Last-Modified'] = Time.now.to_s

answered Apr 20 '12 at 20:02

Exequiel

101
2
5

2

This was the solution for me! Although, I needed to format the time like so: Time.now.ctime.to_s – Matt Fordham Apr 26 '12 at 21:18
I've searched a while to find this response. I don't understand why when you don't specify the header it doesn't stream... anyway, adding this line worked for me. tx – joel1di1 Sep 11 '13 at 12:35

score 5 · Answer 5 · answered Mar 14 '12 at 10:00

Just for the record, rails >= 3.1 has an easy way to stream data by assigning an object that respond to #each method to the controller's response.

Everything is explained here: http://blog.sparqcode.com/2012/02/04/streaming-data-with-rails-3-1-or-3-2/

shuji.koike · Answer 6 · 2014-04-14T17:44:04.000

2

In addition, you will have to set the 'Content-Length' header by your self.

If not, Rack will have to wait (buffering body data into memory) to determine the length. And it will ruin your efforts using the methods described above.

In my case, I could determine the length. In cases you can't, you need to make Rack to start sending body without a 'Content-Length' header. Try to add into config.ru "use Rack::Chunked" after 'require' before the 'run'. (Thanks arkadiy)

edited Apr 14 '14 at 17:44

answered Jun 27 '12 at 22:59

shuji.koike

151
1
4

If you do not know the the length you may try to add into config.ru "use Rack::Chunked" after 'require' before the 'run' – arkadiy kraportov Sep 25 '12 at 05:13

score 2 · Answer 7 · answered Oct 08 '10 at 01:12

2

Yes, response_body is the Rails 3 way of doing this for the moment: https://rails.lighthouseapp.com/projects/8994/tickets/4554-render-text-proc-regression

answered Oct 08 '10 at 01:12

Daniel Cadenas

63
6

score 2 · Answer 8 · answered Aug 30 '11 at 23:28

This solved my problem as well - I have gzip'd CSV files, want to send to the user as unzipped CSV, so I read them a line at a time using a GzipReader.

These lines are also helpful if you're trying to deliver a big file as a download:

self.response.headers["Content-Type"] = "application/octet-stream" self.response.headers["Content-Disposition"] = "attachment; filename=#{filename}"

score 1 · Answer 9 · answered Jan 16 '13 at 06:11

Applying John's solution along with Exequiel's suggestion worked for me.

The statement

self.response.headers['Last-Modified'] = Time.now.to_s

marks the response as non-cacheable in rack.

After investigating further, I figured one could also use this :

headers['Cache-Control'] = 'no-cache'

This, to me, is just slightly more intuitive. It conveys the message to any1 else who may be reading my code. Also, in case a future version of rack stops checking for Last-Modified , a lot of code may break and it may be a while for folks to figure out why.

score 1 · Answer 10 · answered Nov 09 '10 at 14:19

1

I commented in the lighthouse ticket, just wanted to say the self.response_body = proc approach worked for me though I needed to use Mongrel instead of WEBrick to succeed.

Martin

answered Nov 09 '10 at 14:19

Martin

11
1

Ruby on Rails 3: Streaming data through Rails to client

10 Answers10

Linked