Should I use CFTHREAD to slow down bot traffic?

Question

We recently migrated our site from application.CFM to application.CFC. The CFM version could handle our excessive bot traffic, but our CFC version can't. We are trying to figure out why the CFC problem is. In the meantime, we are trying to limit bot traffic.

Currently, I am looking for a solution within the code base to slow bot traffic. We can do this by looking at the user agent as well as IP address.

We have used the code below to successfully stop many bots.

<cffunction name="OnRequestStart">
    <cfif find("bot", cgi.httP_USER_AGENT)>
       <cfabort>
    </cfif>
</cffunction>

Obviously, we do want some bot traffic. But right now, we can't handle all of the bot traffic. It appears that as soon as we you abort to stop a request, another request is right behind it and eventually they bring down our server.

Instead of stopping the bots, what would the ramifications be of using CFTHREAD to slow the bots?

<cffunction name="OnRequestStart">
    <cfif find("bot", cgi.httP_USER_AGENT)>
        <cfthread action="sleep" duration="5"></cfthread>
    </cfif>
</cffunction>

Would using CFTHREAD just stack up the requests and eventually kill our server or would the bots respond with fewer requests per hour?

or you could put limiters farther upstream, e.g. at the firewall level, to keep the bots from ever reaching the CF layer. — Marc B, Jun 30 '14 at 15:18
What are the details of being unable to handle the bot traffic? — Dan Bracuk, Jun 30 '14 at 15:22
Yes, we could and should and will limit bots above the CF layer in a number of ways. I understand that it's IIS's job to handle this type of issue. Still, can I use CFTHREAD to slow specific user agents and what are the ramifications of doing so? — Evik James, Jun 30 '14 at 15:28
I don't really know the answer, however you could try a service like: https://loader.io/ and run a test yourself to see what happens. — Jarede, Jun 30 '14 at 16:13
Are the bots creating too many sessions, which is crashing the JVM? You can change the session timeout for bot connections to one second so that the bot sessions don't accumulate and crash the JVM. — Carl Von Stetten, Jun 30 '14 at 19:49
Carl, that is an excellent question. When we allow bots, it overloads memory and the server spirals out of control within a half hour. — Evik James, Jun 30 '14 at 19:52
Which web server are you using? We use IIS 7.5 and use the URL Rewrite module with a couple user-agent rules to return a 403 response for known abusive user agents. (We do this so that ColdFusion isn't even involved in the blocking process.) — James Moberg, Jun 30 '14 at 20:24
@EvikJames, as always, Ben Nadel has the answers... http://www.bennadel.com/blog/1083-coldfusion-session-management-and-spiders-bots.htm and http://www.bennadel.com/blog/1847-explicitly-ending-a-coldfusion-session.htm — Carl Von Stetten, Jun 30 '14 at 20:44
Tangential to your question, but I sorely doubt it's a difference between Application.cfm and Application.cfc that's causing this. I recommend you post both to http://codereview.stackexchange.com/ so we can have a look at what yer doing. — Adam Cameron, Jun 30 '14 at 22:25
@JamesMoberg We are using this technique too, especially for the bot 80Legs which crashed our web server. The sessionTimeout for all other bots is set to 10 seconds, based on the article of Ben Nadel mentioned by Carl Von Stetten. These two solutions combined have increased our servers performance and stability. — Nebu, Sep 17 '14 at 08:04
@EvikJames Don't use cfthread for this. You are just building up requests for the bots. It will most likely slow down your server and not help you at all. — Nebu, Sep 17 '14 at 08:17

Should I use CFTHREAD to slow down bot traffic?

0 Answers0