Rails 3 - cache web service call

Question

In my application, in the homepage action, I call a specific web service that returns JSON.

parsed = JSON.parse(open("http://myservice").read)
@history = parsed['DATA']

This data will not change more than once per 60 seconds and does not change on a per-visitor basis, so i would like to, ideally, cache the @history variable itself (since the parsing will not result in a new result) and auto invalidate it if it is more than a minute old.

I'm unsure of the best way to do this. The default Rails caching methods all seem to be more oriented towards content that needs to be manually expired. I'm sure there is a quick and easy method to do this, I just don't know what it is!

Lars Haugseth · Accepted Answer · 2013-05-26T04:22:25.637

You can use the built in Rails cache for this:

@history = Rails.cache.fetch('parsed_myservice_data', :expires_in => 1.minute) do
  JSON.parse connector.get_response("http://myservice")
end

One problem with this approach is when the rebuilding of the data to be cached takes quite a long time. If you get many client requests during this time, each of them will get a cache miss and call your block, resulting in lots of duplicated effort, not to mention slow response times.

EDIT: In Rails 3.x you can pass the option :race_condition_ttl to the fetch method to avoid this problem. Read more about it here.

A good solution to this in previous versions of Rails is to setup a background/cron job to be run at regular intervals that will fetch and parse the data and update the cache.

In your controller or model:

@history = Rails.cache.fetch('parsed_myservice_data') do
  JSON.parse connector.get_response("http://myservice")
end

In your background/cron job:

Rails.cache.write('parsed_myservice_data',
  JSON.parse connector.get_response("http://myservice"))

This way, your client requests will always get fresh cached data (except for the first request if the background/cron job hasn't been run yet.)

Excellent, this works well. Note that it's :expires_in, not :expire_in. That edit is not significant enough for SO to let me make it (very bizarre system). — tkrajcar, Aug 30 '11 at 17:38

score 1 · Answer 2 · answered Aug 30 '11 at 05:00

1

I don't know of an easy railsy way of doing this. You might want to look into using redis. Redis lets you set expiration times on the data you store in it. Depending on which redis gem you use it'd look something like this:

@history = $redis.get('history')

if not @history
  @history = JSON.parse(open("http://myservice").read)['DATA']
  $redis.set('history', @history)
  $redis.expire('history', 60)
end

Because there's only one redis service this will work for all your rails processes.

answered Aug 30 '11 at 05:00

chrismealy

4,830
2
23
24

Or you could just store it in your SQL database. – chrismealy Aug 30 '11 at 05:07
interesting thought, I had not considered using redis. Installing a separate cached server just to cache one array is not realistic for this project, but +1 anyway :) – tkrajcar Aug 30 '11 at 17:39
I upvoted Lars's answer too. If somebody did want to use redis, they should probably just use it with Rails.cache and the `redis-store` gem. `:expire_in` works with all the cache stores with rails 3. There's also just plain old memcache (see http://stackoverflow.com/questions/4221735/rails-and-caching-is-it-easy-to-switch-between-memcache-and-redis and http://stackoverflow.com/questions/4188620/redis-and-memcache-or-just-redis). – chrismealy Aug 30 '11 at 18:24

dexter · Answer 3 · 2011-08-30T08:21:03.297

We had a similar requirement and we ended up using Squid as a forward proxy for all the webservice calls from the rails server. Squid was configured to have a cache-expiry time of 60 seconds.

http_connection_factory.rb:

class HttpConnectionFactory
  def self.connection
    AppConfig.use_forward_proxy ? Net::HTTP::Proxy(AppConfig.forward_proxy_host, AppConfig.forward_proxy_port) : Net::HTTP
  end
end

In your application's home page action, you can use the proxy instead of making the call directly.

connector = HttpConnectionFactory.connection 
parsed = JSON.parse(connector.get_response("http://myservice"))
@history = parsed['DATA']

We had second thoughts about using Redis or Memcache. But, we had several service calls and wanted to avoid all the hassles of generating keys and sweeping them at appropriate times.

So, in our case, the forward proxy took care of all those nitty gritties. Please refer to Squid Wiki for the configuration parameters necessary.

Rails 3 - cache web service call

3 Answers3