0

So, I have an eshop (php), and noticed that some competitors are always changing their prices to be lower that mine. I think that they are crawling my website for prices. Is there any way to stop that?

I think that one of the solutions would be to block crawlers IP, but I don't know which IP it is.

Could I block it if there are too many request?

miken32
  • 42,008
  • 16
  • 111
  • 154
Aleksandr
  • 91
  • 10
  • Use for example GoAccess to analyze you web logs, and then throw 404 to crawlers :) – Itsmeromka Oct 26 '15 at 12:48
  • You can try to block by invalid user-agents, but that is a losing battle. Any web client can change the user agent. Also, what makes you think they are crawling your website? – JSK NS Oct 26 '15 at 12:49
  • 4
    Just check you access logs - any scrapeing activity should be easy to spot. Then bad the associated IPs. However the real problem is a business model one, not a programming one. – Steve Oct 26 '15 at 12:49
  • 2
    Crawl the competitor's website in return :) – Oleg Oct 26 '15 at 13:36
  • 1
    Possible duplicate of [Detecting 'stealth' web-crawlers](http://stackoverflow.com/questions/233192/detecting-stealth-web-crawlers) – chishaku Oct 26 '15 at 20:14

1 Answers1

0

Think about enabling reCAPTCHA for your Website, see e.g. https://www.google.com/recaptcha/intro/index.html