Rails + thin: Not possible to download large files

Question

I have a rails app where users can manage large files (currently up to 15 GB). They have also the possibility to download the stored files.

Everything works perfect for files < 510 MB. But for > 510 MB, the download stops after 522,256 KB (510 MB).

I think thin produces this issue. When I start my dev server using thin, I cannot download the complete file. When I start the dev server using webrick, everything works.

I used top to compare the RAM/CPU behavior, but both server, thin and webrick, behave the same way. In development, both server read the complete file into RAM and then send it to the user/client.

I tried to change some options of send_file like stream, or buffer_size. I also set length manually. But again, I was not able to download the complete file using thin.

I can reproduce this behavior using Firefox, Chrome, and curl.

The problem is that my productive rails app uses 4 thin servers behind an nginx proxy. Currently, I cannot use unicorn, or passenger.

In development, I use thin 1.6.3, rails 4.1.8, ruby 2.1.2.

def download
  file_path = '/tmp/big_file.tar.gz' # 5 GB
  send_file(file_path, buffer_size: 4096, stream: true)
end

What have you observed happening? I didn't see support for 'stream' in the send_file API: http://apidock.com/rails/ActionController/Streaming/send_file But there was some support for X-SendFile, which might be the more appropriate choice. Thin supports Rack, which in turn provides the X-SendFile translation for the web server. — Berin Loritsch, Jan 20 '15 at 12:35
@BerinLoritsch Ups, `stream` is something old. I also tried `x_sendfile` without effect. The download stops and Chrome, or Firefox 'thinks' the file was successfully downloaded. — Bjoernsen, Jan 20 '15 at 12:51

score 1 · Accepted Answer · edited May 23 '17 at 12:20

If you are using send_file, it is ideal to use a front end proxy to pass off the responsibility of serving the file. You said you are using nginx in production, so:

In your production.rb file, uncomment config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'.

You will also have to change your nginx configuration to accommodate Rack::Sendfile. Its documentation is located here. The changes amount to adding:

proxy_set_header X-Sendfile-Type X-Accel-Redirect;
proxy_set_header X-Accel-Mapping /=/files/; # or something similar that doesn't interfere with your routes

to your existing location block and adding an additional location block that handles the X-Accel-Mapping that you added. That new location block might look like:

location ~ /files(.*) {
  internal;             
  alias $1;             
}

You will know it is working correctly when you ssh to your production server and curl -I the thin server (not nginx) and see a X-Accel-Redirect header. curl (no -I) directly to the thin server should not send the file contents.

You can see my recent struggle with nginx and send_file here.

Rails + thin: Not possible to download large files

1 Answers1