12

I have a path (mysite.com/myapiendpoint for sake of example) that is both resource intensive to service, and very prone to bot abuse. I need to rate limit access to that specific path to something like 10 requests per minute per client IP address. How can this be done?

I'm hosting off an EC2 instance with CloudFront and AWS WAF in front. I have the standard "Rate Based Rule" enabled, but its 2,000 requests per minute per IP address minimum is absolutely unusable for my application.

I was considering using API Gateway for this, and have used it in the past, but its rate limiting as I understand it is not based on IP address, so bots would simply use up the limit and legitimate users would constantly be denied usage of the endpoint.

My site does not use sessions of any sort, so I don't think I could do any sort of rate limiting in the server itself. Also please bear in mind my site is a one-man-operation and I'm somewhat new to AWS :)

How can I limit the usage per IP to something like 10 requests per minute, preferably in WAF?

[Edit]

After more research I'm wondering if I could enable header forwarding to the origin (running node/express) and use a rate-limiter package. Is this a viable solution?

Community
  • 1
  • 1
user5071535
  • 1,312
  • 8
  • 25
  • 42
  • We would also like to do something similar to this and could forward and check the IP in our application but this would massively reduce the benefits of using CloudFront as a CDN as we cache across users – Sam Collins Oct 24 '18 at 12:16
  • Just to follow up on this, I ended up using express-rate-limit to limit access to certain API endpoints to only a few requests per minute per IP address. A WAF rule would be a better option though for various reasons – user5071535 Jan 16 '19 at 03:45

3 Answers3

17

I don't know if this is still useful to you - but I just got a tip from AWS support. If you add the rate limit rule multiple times, it effectively reduces the number of requests each time. Basically what happens is each time you add the rule, it counts an extra request for each IP. So say an IP makes a single request. If you have 2 rate limit rules applied, the request is counted twice. So basically, instead of 2000 requests, the IP only has to make 1000 before it gets blocked. If you add 3 rules, it will count each request 3 times - so the IP will be blocked at 667 requests.

The other thing they clarified is that the "window" is 5 minutes, but if the total is breached anywhere in that window, it will be blocked. I thought the WAF would only evaluate the requests after a 5 minute period. So for example. Say you have a single rule for 2000 requests in 5 minutes. Say an IP makes 2000 requests in the 1st minute, then only 10 requests after that for the next 4 minutes. I initially understood that the IP would only be blocked after minute 5 (because WAF evaluates a 5 minute window). But apparently, if the IP exceeds the limit anywhere in that window, it will be locked immediately. So if that IP makes 2000 requests in minute 1, it will actually be blocked from minute 2, 3, 4 and 5. But then will be allowed again from minute 6 onward. This clarified a lot for me. Having said that, I haven't tested this yet. I assume the AWS support techie knows what he's talking about - but definitely worth testing first.

Luke
  • 173
  • 1
  • 6
  • 1
    This sounds like a pretty good compromise if it's true. Can anyone vouch for this or point out some documentation on it? – user5071535 Jan 16 '19 at 03:38
  • I can verify that we have used this "feature" as well. It's not documented, and it may actually be a bug, but I'll say that this "stacking" works for us. – Nicolas Webb Apr 03 '19 at 17:08
4

AWS have now finally released an update which allows the rate limit to go as low as 100 requests every 5 minutes.

Announcement post: https://aws.amazon.com/about-aws/whats-new/2019/08/lower-threshold-for-aws-waf-rate-based-rules/

Sam Collins
  • 443
  • 5
  • 13
  • Finally! I was using Luke's solution plus rate limiting in my node server, but this is a nice improvement – user5071535 Aug 31 '19 at 19:03
  • The Catch here is that this works only through their web console as of today. API and CLI and are still locked at 2000 for min value. – Praveen Sastry Oct 02 '19 at 06:07
-2

Using rule twice will not work, because WAF rate based rule will count on cloud watch logs basis, both rule will count 2000 requests separately, so it would not work for you. You can use AWS-WAF automation cloud front template, and choose lambda/Athena parser, this way, request count will perform on s3 logs basis, also you will be able to block SQL,XSS and bad bot requests.

Jay seen
  • 493
  • 4
  • 14