1

Hi I've recently been getting super high spikes in Apache CPU usage, Apache memory usage and MySQL memory usage. It turns out that crawlers were accessing my site at a very aggressive rate, specifically Facebook. I attempted to add a crawl delay for the facebook crawler to the robot.txt file, as seen below:

User-agent: Facebot
Disallow:
Crawl-delay: 5

but I'm still seeing high usage spikes. Is this due to incorrect code or is there a better way entirely to go about stopping crawlers using so much of my server's resources?

Any help is much appreciated.

Jake Jacobs
  • 33
  • 4
  • It seems this is still a current issue and FB crawler does not always play nicely https://stackoverflow.com/questions/11521798/excessive-traffic-from-facebookexternalhit-bot – SolaceBeforeDawn Dec 12 '18 at 03:24

1 Answers1

0

Change the path to

 Disallow: /

see Robots FAQ

Julien Nioche
  • 4,772
  • 1
  • 22
  • 28