0

Facebook Crawling My server using that range of IP Addresses - That cause my server down for many times

31.13.97.117
31.13.97.116
31.13.97.113
31.13.97.115
31.13.97.112
31.13.97.118
31.13.97.114
31.13.97.119

First: its regardless any Robots.txt Rules... so can not using Crawler-Delay with it. Second: stop the crawling process using iptables (the linux firewall) cause problem on OpenGraph reading, likes, sharing, and comments of Facebook social plugin.

please how to stop this crawling or control them with keeping social plugins works.. ?

I see this solution: excessive traffic from facebookexternalhit bot

but can the solution be Apache Webserver or Server side?

Thanks and Regards

Community
  • 1
  • 1
user27093
  • 1
  • 2

1 Answers1

0

I decide as fast solution to use IPtables (firewall) to make rate or connection limits for this range of IPS So avoid any programming throttle way.

now every 60 sec can do 10 connections only, and this is still in test.

my IPtables Rules Exactly:

iptables -I INPUT -p tcp --dport 80 -i eth0 -m iprange --src-range 31.13.64.1-31.13.127.255 -m state --state NEW -m recent --set

iptables -I INPUT -p tcp --dport 80 -i eth0 -m iprange --src-range 31.13.64.1-31.13.127.255 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 -j DROP

note that; it must be adjustment for you case, and according to your rating...

because they can affect Sharing,Likes,Comments, and so most of Facebook plugins.

user27093
  • 1
  • 2