25

So I have turned this on in my Azure cloud service site to try mitigate spiders and bots hitting us an absurd amount of times.

Has anyone had any experience with these settings?

Deny IP address based on the number of concurrent request: Maximum number of concurrent requests?

Deny IP address based on the number of requests over a period of time: Maximum number of request? Time period (in milliseconds)?

dimoss
  • 1,506
  • 2
  • 16
  • 24
  • 3
    Why do you want to block spiders? Your service should be designed that spiders (or any user, for that matter) should cause performance problems. I think you're approaching the problem by treating the symptom, not the cause. – Dai Feb 11 '13 at 02:39
  • 3
    Because we have 10's of thousands of products and rogue spiders constantly hit all our product pages. I measured 27,000+ hits from 1 particular spider in a few hours. Not to mention malicious spiders that hit the pages an absurd amount of times looking for vulnerabilities, plus attempted DOS attacks which has happened in the past. A normal user or good bot/spider would not be hitting sites this absurd amount of times from a single IP. FYI the site is going fine but this is a feature in IIS8 and would like to hear peoples experiences if they use it. – dimoss Feb 12 '13 at 01:40
  • 2
    I have a different use case. Denying fraudsters who are hitting an e-commerce site. If I allow them free access, they do not behave like the bots you described. They will attempt a transaction every minute or so. I'd like to deny them any more than 5 tries. Since each page generates up to 6 GETs, I want to try: "Deny IP Address based on the number of requests over a period of time", and setting it at something like: "Maximum number of requests" = 30, "Time period in milliseconds" = 86400000 (24 hours). But something tells me putting a huge number in this last slot will break the tool. – Mark Goldfain Dec 31 '14 at 23:29
  • 1
    For such a major feature of IIS, there is little or no REAL WORLD usage information. I have started a bounty on this question to get more views. – Muhammad Rehan Saeed Apr 15 '15 at 20:34
  • See here for more answers: http://serverfault.com/questions/416233/dynamic-ip-restriction-strategy – Keith Mar 09 '16 at 16:03

2 Answers2

22

An experimental approach to find sensible settings

I have recently been experimenting with these settings to decide on values for our production site.

We determined the maximum number of requests a single (request-heavy) page generates, and multiplied that with 2.5 to get the maximum number of requests over time. For the time value I chose 200ms.

Manual testing shows that these settings work fine for "normal" usage. We manage to get some 403 Forbidden when we simultaneously reload the page in 5 or more tabs in the browser.

Something you have to keep in mind is that many users of your website may be sitting behind the same proxy, so the Dynamic IP Restrictions consider these users as only one. With the rather short window of 200ms I expect that this will not be a problem, while still blocking aggressive DoS attacks to some extent.

Also, we do not restrict the number of simultaneous connections. It is nearly impossible to find a sensible number here, as the number of different clients is potentially unbounded.

Note that the requests a single page (take the one with most requests) is highly relevant to get to useful settings. E.g. if a page load of your front page generates 10 requests to your server, these will come in a very short time span, so your restrictions must have a higher threshold.

Update April 2015

Our service has been running for over a year with these settings, and we've been quite happy so far.

theDmi
  • 17,546
  • 6
  • 71
  • 138
  • 1
    Very sensible approach, just realized a lot of our clients are corporate and thus of course share the same IP. Any adjustments you'd suggest based on traffic ..ie unique simultaneous connected users ? – Robert Hoffmann Jan 24 '16 at 22:39
  • @Robert thats a question that came to my mind at first. What did you do about it? – yakya May 16 '16 at 08:57
  • 1
    regarding the "same proxy" issue - please note that the latest version of this module (built-in into iis8) has the "proxy mode" that looks at the "X-Forwarded-For" header rather than the actual IP address. https://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-dynamic-ip-address-restrictions – jazzcat Mar 02 '17 at 18:24
  • @jazzcat: Clients can trivially modify that header. You should only respect the x-forwarded-for header if the request is being forwarded by a machine you control (e.g., a load balancer). Of course, if you use a load balancer, you would usually want to place your dynamic restrictions on the load balancer itself rather than on IIS. – Brian Apr 05 '19 at 19:05
1

Some people do not have access to a server, or like me they are not satisfacted by the Dynamic IP restriction, so i have made a script for asp classic.

You can place it on the webpage you want (homepage and/or internal). It use a Mysql DB. In the example i have set a ban for each ip loading 3 webpage in 3 seconds (that is not a normal activity). I just wana looking to block every flooding, aspiration script, ddos, bot or annoying access to my website.

  1. YOU NEED TO CREATE A MYSQL DATABASE :
    CREATE TABLE `banip` (
      `id` int(11) NOT NULL auto_increment,
      `IP` char(15) default NULL,
      `dtime` time default NULL,
      PRIMARY KEY  (`id`),
      KEY `IP` (`IP`)
    ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
  1. YOU NEED TO PUT THE ASP CODE WHERE IS NEEDED

  2. PLUS A TINY ADMIN WEBPAGE mybanipadm.asp (can change the filename)

ASP CLASSIC CODE :

<%
' ***PUT THIS CODE AT THE TOP OF YOUR WEBPAGE YOU WANT TO PROTECT***
' COULD BE HOME PAGE AND/OR INTERNAL PAGE
' THE BAN IS PERSISTANT UNTIL THE SERVER RESTART

response.buffer = true
IP = Request.ServerVariables("REMOTE_ADDR")

'IP WHITELIST - SEPARATE EACH IP WITH A |
IPWL = "127.0.0.1|"

if instr(IPWL,IP) then
'do nothing the ip is whitelisted
else

'CHECK IF THERE IS A BAN THAT MATCH THE CURRENT IP
if Application("mybanip") <> "" then
if instr(Application("mybanip"),IP) then

' RESPONSE EXAMPLE WHEN ACCESS DENIED (CHOOSE ONE OR MAKE YOUR)
'Response.Status = "403 Forbidden"
'Response.Status = "404 Not Found"
'response.redirect "banned.html"
response.write "You are going too fast !"

session.abandon
response.end
end if
end if


' THE TIME NOW
dtime = FormatDateTime(now(),3)

'we can decide to run it at speficied time
'if dtime >= "00:00:00" and dtime < "05:00:00" then


' PREPARE TO CHECK DATABASE FOR THE LAST 3 SECONDS ACTIVITY
secfrom = DateAdd("s",-3,now()) 'value you can change is -3 (seconds)
secfrom = FormatDateTime(secfrom,3)

' ***OPEN THE CONNEXION STRING (USE YOUR ONE OR MODIFY THIS)***
Set conn = Server.CreateObject("ADODB.Connection")
conn.Open "DRIVER={MySQL ODBC 3.51 Driver};server=127.0.0.1;uid=LOGIN;pwd=PSW;Database=DBNAME;"

' ***EVERYTHING BELOW MUST BE PUT AFTER THE CONNEXION STRING OPENED***

' POPULATE DATABASE WHIS THE CURRENT IP AND TIME
SQL = "INSERT INTO BANIP (IP,DTIME) values('" & IP & "','" & dtime & "')"
conn.execute(SQL)

' CHECK IF THERE IS A SPAM ACTIVITY FOR THE CURRENT IP
SQL = "SELECT COUNT(IP) as nbfound FROM BANIP WHERE IP='" & IP & "' AND dtime BETWEEN '" & secfrom & "' AND '" & dtime & "'"
set rsIPCount = conn.Execute(SQL)
if not rsIPCount.Eof then
ipcount = clng(rsIPCount("nbfound"))
else
ipcount = "0"
end if
rsIPCount.Close
set rsIPCount = nothing

' IF THERE IS AT LEAST 3 WEBPAGE LOADED IN 3 SECONDS ACTIVITY THEN SET A BAN
if ipcount >= 3 then 'value you can change is 3 (webpage)
application.lock
Application("mybanip") = Application("mybanip") & IP & "|"
application.unlock
end if


' DELETE ALL ENTRY EVERY 2 MINUTES FOR PERFORMANCE
if Application("mybanipdel") = "" then
Application("mybanipdel") = dtime
elseif datediff("n", Application("mybanipdel"), dtime) >= 2 or datediff("n", Application("mybanipdel"), dtime) < 0 then 'value you can change is 2 (minutes)
conn.execute "DELETE FROM BANIP"
Application("mybanipdel") = FormatDateTime(now(),3)
end if

SQL = ""
IP = ""
end if

%>

admin page mybanipadm.asp

<html>
<head>
<title>My admin</title>
</head>
<body><%

if request.querystring("disconnect")="yes" then
session("adm") =""
elseif request.querystring("clear")="yes" then
Application("mybanip") = ""
end if

' ***CHANGE THIS VALUES***
login = "login"
passw = "pass"

if request.form("LogMe")<>"" and (request.form("login")=login and request.form("passw")=passw) then
session("adm") = "loggued"
elseif session("adm") = "" then
response.write "<p>Please log-in :</p> <form method=""post""><input type=""text"" size=""15"" name=""login"" placeholder=""login""> <input type=""password"" size=""15"" name=""passw"" placeholder=""password""><input type=""submit"" name=""LogMe""></form>"
response.end
end if

response.write "<p><a href=""?disconnect=yes"">Disconnect from the admin</a> - <a href=""?clear=yes"">Clear all ip</a></p>"

if request.form("unban")<>"" and request.form("ipban")<>"" then
application.lock
Application("mybanip") = replace(Application("mybanip"),request.form("ipban") & "|","")
application.unlock
response.write "<p>IP : <b>" & request.form("ipban") & "</b> has been unbanned !</p>"
end if

response.write "Unban this IP : <form method=""post""><input type=""text"" size=""15"" maxlenght=""15"" name=""ipban"" placeholder=""000.000.000.000""> <input type=""submit"" name=""Unban"" value=""Unban""></form>" 
response.write "<p>IP CURRENTLY BANNED</p>" & replace(Application("mybanip"),"|","<br>")

%>
</body>
</html>
David
  • 11
  • 2