How to cache files over 1MB with rack/cache on Heroku?

Question

I'm using a combination of Dragonfly and rack/cache hosted on Heroku.

I'm using Dragonfly for uploaded assets. Thumbnails are processed on-the-fly and stored in rack/cache for fast delivery from memcached (via the Memcachier addon).

Regular static assets are also cached in memcached via rack/cache.

My problem is that any uploaded files over 1MB are causing a 500 error in my application.

2013-07-15T10:38:27.040992+00:00 app[web.1]: DalliError: Value too large, memcached can only store 1048576 bytes per key [key: d49c36d5db74ef45e957cf169a0b27b83b9e84de, size: 1502314]
2013-07-15T10:38:27.052255+00:00 app[web.1]: cache: [GET /media/BAhbBlsHOgZmSSIdNTA3Njk3ZWFiODBmNDEwMDEzMDAzNjA4BjoGRVQ/WTW_A5Flyer_HealthcareMedicalObsGynae_WEB.pdf] miss, store
2013-07-15T10:38:27.060583+00:00 app[web.1]: !! Unexpected error while processing request: undefined method `each' for nil:NilClass

Memcache has a limit of 1MB, so I can understand why my asset was not cached, but I would rather it didn't break serving assets.

I'm not even sure where this error is coming from. Presumably from one of the other rack middlewares?

Increasing the maximum file size doesn't seem to have have any affect.

config.cache_store = :dalli_store, ENV["MEMCACHIER_SERVERS"].split(","), {¬
  :username        => ENV["MEMCACHIER_USERNAME"],¬
  :password        => ENV["MEMCACHIER_PASSWORD"],¬
  :value_max_bytes => 5242880 # 5MB¬
}

Long term, I know that moving this sort of asset off of Heroku is a sensible move, but that won't be a quick job.

What can I do to serve these assets on Heroku in the meantime without errors?

score 10 · Answer 1 · answered Sep 17 '13 at 22:26

10

So contrary to @jordelver's question, I find that setting the :value_max_bytes option of dalli does work. I'm setting up Rack::Cache in a slightly different way that maybe makes the difference.

This is what my production.rb contains to configure Rack::Cache:

client = Dalli::Client.new(ENV["MEMCACHIER_SERVERS"],
                           :username => ENV["MEMCACHIER_USERNAME"],
                           :password => ENV["MEMCACHIER_PASSWORD"],
                           :value_max_bytes => 10485760)
config.action_dispatch.rack_cache = {
  :metastore    => client,
  :entitystore  => client
}
config.static_cache_control = "public, max-age=2592000"

With the above, some errors will be printed to the logs for values over 1MB, but they won't cause a 5xx error for the client, just a cache miss.

P.S I work for MemCachier :) so we're interested in trying to sort this out. Please let me know if it works.

answered Sep 17 '13 at 22:26

David Terei

1,985
16
18

setting a higher value_max_bytes value did not work in my case. – Francois Oct 28 '13 at 01:10
This helped point me in the right direction on a slightly different problem, so thanks David :) – robotmay Nov 18 '13 at 23:46
To do it this way, you have to install rack-cache gem. Normally you turn on dalli in `environments/development.rb` by adding `config.cache_store = :dalli_store`. Can you specify the value_max_bytes here? Thank you – Zack Xu May 20 '16 at 17:06
1

Yes you can, `config.cache_store = :dalli_store, ENV['MEMCACHIER_SERVERS'], { :value_max_bytes => 1048760 }` – David Terei May 25 '16 at 06:20

score 8 · Accepted Answer · edited Aug 03 '14 at 08:32

8

I had the same issue as @jordelver and managed to get round memcachier's limits by monkey patching Dragonfly::Response:

module Dragonfly
  class Response
    private
    def cache_headers
      if job.size > 1048576
        {
          "Cache-Control" => "no-cache, no-store",
          "Pragma" => "no-cache"
        }
      else
        {
           "Cache-Control" => "public, max-age=31536000", # (1 year)
           "ETag" => %("#{job.signature}")
        }
      end
    end
  end
end

Essentially, if the size is over 1048576 bytes, send a no-cache header.

edited Aug 03 '14 at 08:32

BenB

10,300
7
32
30

answered Oct 08 '13 at 15:59

aaronrussell

9,389
5
38
62

1

FYI there's an API for changing headers without monkey-patching - see http://markevans.github.io/dragonfly/configuration/ ("response_header" bit) – Mark Evans Aug 03 '15 at 11:26

Dorian · Answer 3 · 2014-09-15T22:25:26.447

0

My application.js was too big for rack-cache so I did:

# in config/environments/development.rb
config.action_dispatch.rack_cache = {
  metastore: 'file:/var/cache/rack/meta',
  entitystore: 'file:tmp/cache/rack/body'
}

And it works!

Stores metadata in memcache but actual file in filesystem and not in memory.

edited Sep 15 '14 at 22:25

answered Sep 12 '14 at 17:47

Dorian

22,759
8
120
116

1

This isn't a good idea if you're running on Heroku as Dynos will share the metastore but not the filesystem. This means that sometimes your file will be cached according to the metastore but missing from the filesystem and it won't be found. – jordelver Sep 12 '14 at 22:14
Do you recommend storing files in Memcache? – Dorian Sep 13 '14 at 17:15
And BTW, this is for `development.rb` – Dorian Sep 13 '14 at 17:16
I changed to use local file system for meta store too – Dorian Sep 15 '14 at 22:25

How to cache files over 1MB with rack/cache on Heroku?

3 Answers3

Linked