2

I had a recent outage on a Nginx/Rails application server. It turned out we were being bombarded by requests to a particular URL that takes a few seconds to load. It appears that a user was continually refreshing that page for a number of minutes - my guess is they accidentally put some object on their keyboard in such a way as to trigger a constant stream of browser refreshes.

Regardless of the cause, I need to put protection in place against this kind of problem, and note that this is not static content - it's dynamic, user-specific content sitting behind authentication.

I've looked into using Cache-Control but this appears to be a non-starter - on Chrome at least, refreshing a page within the same tab will trigger a request regardless of the Cache-Control header (cf iis - Is Chrome ignoring Cache-Control: max-age? - Stack Overflow)

I believe the answer may be rate limiting. If so, I wouldn't be able to do it based on IP because many of our customers share the same one. However I may be able to add a new header to identify a user and then apply rate limiting in Nginx based on this.

Does this sound like the way forward? This feels like it should be a fairly common problem!

griswoldbar
  • 499
  • 3
  • 10
  • 2
    Cache on the rails side. There's no reason you can't cache the response or the underlying data based on the page parameters for a set interval so you can skip the expensive processing no matter how many times they refresh. – dbugger Dec 14 '21 at 12:30
  • 2
    This is a very common scenario known as a Dedicated Dential of Service attack. Rate limiting in Rack applications (like Rails) can be done with the [`Rack::Attack`](https://github.com/rack/rack-attack) gem. But its very advantageus to doing it on the layer above it (such as NGinX) if possible. If you do it on the Rack layer each request is still hitting your Rails server. – max Dec 14 '21 at 12:46
  • Caching doesn't quite work here - its really up to the client to "respect" cache-control headers and so it provides no protection against a DDOS attack by a bad actor. The exception is a reverse cache which mitigates the effects somewhat as the requests will just hit your cache instead of your app but its only suitible if the content is cacheable. – max Dec 14 '21 at 12:51
  • Given its a authenticated user causing the issue, going to rate limiting seems overkill in this situation. And you don't necessarily have to cache the entire content -- just caching the underlying data representation is often sufficient. – dbugger Dec 14 '21 at 12:55

2 Answers2

1

Nginx rate limiting is a fast configuration update if immediate mitigation is needed. As others have mentioned, caching would also be ideal when combined with this.

server {
  # DoS Mitigation - Use IP and User Agent to prevent against NAT funnels from different computers
  limit_req_zone $host$binary_remote_addr$http_user_agent zone=rails_per_sec:10m rate=2r/s;

  upstream rails {...}

  try_files $uri $uri/ @rails;

  location @rails {
    limit_req zone=rails_per_sec burst=10 nodelay;
    ...
  }
}

The $http_authorization header or a unique cookie (e.g. $cookie_foo) could also be used to uniquely identify requests that would collide with the same IP/user-agent values.

limit_req_zone $host$binary_remote_addr$http_authorization  ...;
limit_req_zone $host$binary_remote_addr$cookie_foo          ...;
anthumchris
  • 8,245
  • 2
  • 28
  • 53
0

A colleague of mine suggested a solution that I think is the best fit for our situation. I'll explain why in case this proves useful to anyone else.

Note that we were receiving requests at a low rate - just 6 per second. The reason this was a problem was that the page in question was quite a slow loading report, only accessible to authenticated users.

Server-side caching is not a great solution for us because it needs to be implemented individually on each affected page and we have a complex app with lots of different controllers.

Rate-limiting via Nginx might be viable but tricky to optimise and also has issues with testability.

Anyway, my colleague's solution is as follows: we already have a table that logs details of each request, including the ID of the user that made it. To find out if a user is refreshing too often, we simply schedule a Sidekiq job once every, say, 30 seconds to check this table for users with a refresh rate above our threshold and then kill any active sessions.

How you kill a session depends how you are managing them - in our case, we could simply add a flag to the user that says "rate_limited" and have our Sidekiq job set it to true, and then check the value of this flag on each request. If it's true, the user will be redirected away from the slow page and on to the login screen which will happily deal with refreshing itself 6 times per second.

You could achieve something similar even without a request logging table, e.g. by keeping track of the request rate in a new column on the users table.

Note that this solution is a better UX than Nginx rate-limiting, as users are never actually locked out of the app.

griswoldbar
  • 499
  • 3
  • 10