13

Our production server has been producing invalid authenticity token errors for several months now. The errors are produced on almost all forms sending (PUT|POST|DELETE) requests. Sometimes the error occurs, sometimes they don't. There appears to be no rhyme or reason as to why they occur. The error itself does not occur often but it is a worry for us. Below is an example of what a typical form that causes this error looks like.

<form class="button_to" method="post" action="/lesson_progress_trackers/333">  
  <input type="hidden" name="_method" value="patch">
  <input class="finish-lesson-button" type="submit" value="Done!">
  <input type="hidden" name="authenticity_token" value="Qd3FsJZY2UXR9vahuFmaY5rrqA+J5xzGpl4cGI2Vwerx8PZPQtDMugz6oqoe3iviC+/U5zTYPdeX3apwbap09E==">
  <input type="hidden" name="completed" value="true">
</form>

Here's what I've discovered so far.

  1. We use Turbolinks 2.5.3 (we have not updated this in over a year).
  2. In every case of an invalid token error, the user passed an authenticity token to the server, it just ended up being invalid.
  3. We currently use protect_from_forgery with: :exception in our application controller.
  4. The errors started appearing when we pushed a bunch of new code to production several months ago. This new code spans hundreds of files but so far I've found nothing in the code that would be relevant to this issue.
  5. The error can occur on any type of browser and device.
  6. There is no correlation between increased traffic and the invalid auth tokens appearing.
  7. Users can come from any country.
  8. These are not bots experiencing these issues. We even had a colleague experience this error though they can't recall what they did to produce it.
  9. The users follow typical if not expected behavior. They are using the app as intended. I looked through their clicks and recorded behavior history to conclude this.

Ultimately I want to figure out how to solve this. My first step is to reproduce the error successfully, but I can't even do that. My question is this: what can I do to get me on my way to figuring out what's causing this? I am running out of options. Thanks!

thank_you
  • 11,001
  • 19
  • 101
  • 185
  • Yes, I did that. This is occurring to real users. These aren't bots. The patterns I've discovered are typical user behavior. There is nothing abnormal in what they are doing. I'm willing to say this is expected behavior. This is why it's so frightening. – thank_you Nov 07 '16 at 14:06
  • Can you rule out that anything expires? Cookies are just one thing, it could be some time-related comparison, wrong time in the server, database field that expects specific time or anything like that. – Smar Nov 07 '16 at 14:13
  • I don't think so? Can you clarify your question a bit. – thank_you Nov 07 '16 at 14:22
  • I’m just trying to provide some ideas you can check, sometimes even vaguely remote things can affect in strange ways. Disk space running out? Wrong time on the server is often cause when doing validity checks since those often have time bindings. Depending of the store for your session data, the actual thing to check may vary, have you tried to swap the session store to another to see if it mitigates the problem? – Smar Nov 07 '16 at 14:28
  • Disk space isn't running out. I'm not aware of our server having an incorrect time. We've been experiencing some memory issues but what I've found is that the memory issues will occur at different times from the invalid auth tokens. I haven't tried replacing the session store. I should also mention I'm using Heroku. – thank_you Nov 07 '16 at 14:38
  • It occurs for me too in Rails 3.2 during signing in. If I have logged multiple times on the same browser without closing the window, I start getting invalid authenticity token error and because of which the sign in fails. Then I just close the window and sign in again and it works. This happens every time. Here's [the question](http://stackoverflow.com/q/39970750/3863146) which I posted but got no answer till now. – Sahil Nov 09 '16 at 04:36
  • 2
    I got this error when I cached authenticity_token. Try to check all `cache` methods in your project. There's probably the same issue. – itsnikolay Nov 13 '16 at 20:43
  • Are you using NGINX on your server? – Jorge Cuevas Jan 16 '17 at 00:56
  • I assume NGINX. I'm using Heroku at this time. – thank_you Jan 16 '17 at 15:11
  • I know it's been a while but any update on this issue? We're experiencing the same pattern the only difference is that we're not on Heroku. – coding addicted Dec 23 '19 at 14:05
  • @codingaddicted Unfortunately no. Best of luck in figuring it out. – thank_you Jan 17 '20 at 22:03

1 Answers1

3

Dunno if this is too late to be useful, but I had the same problem. I was able to reproduce by:

  1. Make sure you are signed out of the app
  2. open a browser tab to the sign in page
  3. Let it sit long enough to expire the session/csrf token (could be several hours)
  4. open another tab to the sign-in page, and log in
  5. go back to the old tab and try to log in again - The InvalidAuthenticityToken exception occurs.

I think that this happened for me because the two tabs shared a single session, the session that was created when the new tab was opened. However, the old tab still had the csrf token from the old session in the login form. When the new session cookie and the old csrf token were submitted together, they did not match and therefore the error is thrown.

I'm not sure how to actually fix this, other than handling the error more gracefully so that the user doesn't see a confusing error page.

BTW, I am using devise, but I don't think that it is specific to Devise.

blaedj
  • 322
  • 2
  • 9