9

We are seeing an unfortunate and likely browser-based CSRF token authenticity problem in our Rails 4.1 app. We are posting it here to ask the community if others are seeing it too.

Please be aware that most error reporting tools — like Honeybadger — automatically suppress ActionController::InvalidAuthenticityToken, so you don't normally see the problem in your error reporting tool unless you go out of your way to see it.

Here's the problem, and this is NOT a development issue — it is a production issue that has yet to be diagnosed.

The exception we see is simply ActionController::InvalidAuthenticityToken on normal logins to our website. Upon careful examination of the authenticity_token sent by the form and the session's _csrf_token (we are using active_record_store as our session_store setting), they just don't match. Upon direct examination, I can conclude only that they are completely different tokens, but I don't know why.

We see this problem broadly, maybe about 1-2% of our high traffic website. I see it only in Production, I am unable to reproduce it in development whatsoever.

I see it on IE 11 and Edge browsers most (you will note Rails 4.1 was released before IE 11 and Edge), but also on Chrome on Android and occasionally mobile Safari too.

Our Cache-control headers are set as follows:

Cache-Control: max-age=0, private, must-revalidate

Jason FB
  • 4,752
  • 3
  • 38
  • 69
  • I've seen plenty of these errors in our application, especially after the 3.2 -> 4.1 upgrade. We use the cookie store rather than active record. Nevertheless, both approaches use a cookie that can have an expiration, which was the root of the problem in our case: the user's browser had a stale form referencing an expired session cookie. We did see a lot more of these errors on mobile devices due to the way android and safari were caching background pages. – Aaron Breckenridge Jul 26 '17 at 14:53
  • Has this been addressed in Rails 5? Indeed, what you say above sounds correct (browsers holding onto cached pages). However, this means the browser is not respecting our Cache-control header which is set explicitly as Cache-Control:no-cache, no-store, max-age=0, must-revalidate – Jason FB Aug 07 '17 at 17:29
  • Actually let me correct that, on the signin page where we see this most the cache-control headers are Cache-Control: max-age=0, private, must-revalidate – Jason FB Aug 07 '17 at 17:46
  • From what I remember, some mobile devices were not obeying the cache-control headers, the biggest problem child being iOS, but Android occasionally too. You may not like the solution, but we ended up bypassing the authenticity token check on sign in. – Aaron Breckenridge Aug 07 '17 at 19:26
  • 1
    The devise docs do reference a difference in behavior in Rails 5 https://github.com/plataformatec/devise/blob/a62faa2c8258ba1c35fe84e147f42c35eccccb8a/README.md#controller-filters-and-helpers. I have little experience with devise though. – Aaron Breckenridge Aug 07 '17 at 19:27
  • I'm gonna change the whole website to Cache-Control:no-cache, no-store, max-age=0, must-revalidate and also Pragma: no-cache (which is supposed to only be for HTTP1.0 browsers, but supposedly is still needed to force modern browsers not to cache). Will deploy these fixes and post here again if it works. Notice current headers on the pages are Cache-Control: max-age=0, private, must-revalidate with no "Pragma" header – Jason FB Aug 08 '17 at 20:31
  • thank you for your help @AaronBreckenridge ! – Jason FB Aug 08 '17 at 20:32

1 Answers1

4

This is been identified and fixed. The cache control headers were not set in our Rails 4.1 application, leading to the default headers of

Cache-Control: max-age=0, private, must-revalidate

This header is not strong enough to force browsers to not cache. Thus, the login form and JSON token were being cached by the client browser — notably mobile clients — and returning session_ids that were expired.

To fix:

Set cache-control and pragma header, as such

Cache-Control:no-cache, no-store, max-age=0, must-revalidate

and

Pragma: no-cache

IN rails, add this to your application_controller.rb : 

before_action :set_cache_headers
def set_cache_headers
  response.headers["Cache-Control"] = "no-cache, no-store, max-age=0, must-revalidate"
  response.headers["Pragma"] = "no-cache"
  response.headers["Expires"] = "Mon, 01 Jan 1990 00:00:00 GMT"
end

Should it be global to every action in your app? This is up to you, but you will definitely want to do this on any controller that renders a form, particularly a log-in form, or for any page that renders a JSON token which might expire. So in in modern apps, the short answer is yes.

If you explicitly want to keep your Rails app responses cached you need to figure out how to explicitly expire these CSRF and JSON tokens if embedded.

Note the symptom manifests at subtle occurrence levels on mostly mobile clients.

I explored this in a blog post here, please visit my blog and consider leaving a comment there to discuss: https://blog.jasonfleetwoodboldt.com/2017/09/03/the-great-rails-cache-lie/

Jason FB
  • 4,752
  • 3
  • 38
  • 69
  • 5
    For anybody else stumbling across this as a usable answer it is worth understanding the basis for this. Per this thread: https://github.com/rails/rails/issues/21948 mobile clients often will kill tabs to free memory, which then appear open and available to submit, but have lost the appropriate cookie to match the CSRF token. Killing the cache completely as this does helps, as long as site performance with no caching does not suffer. – Phil Jan 25 '18 at 09:58
  • Phil -- thank you so much, never knew that about killing tabs. I think browser-side caching for most dynamic web pages in 2018 is a relic from the past. That is, cache your images, JS, css, etc, but why cache a dynamic web page that is going to change every week? The problem is the archaic mentality that "browser caching is good." It's how the web worked in 2008 but not 2018. I recommend serving small, fast pages from Rails (uncached) and then have everything else pulled from a CDN (cached). This is fast & performant. – Jason FB Feb 07 '18 at 14:30
  • @Phil--- thank you for linking the Rails issue I never saw this! – Jason FB Feb 04 '22 at 19:32