How to stop all search engines, bots to crawl some urls

Question

I want to count ads click on a widget.

I've used on robot.txt file:

User-agent: *
Allow: /
Disallow: */ads_count/*

I've also add nofollow for all links in that widget.

But many bots still follow urls in that widget. I've get client ip to count urls and i got many IP form bots.

score 2 · Accepted Answer · answered Dec 04 '12 at 10:43

2

Did u try removing the (*) before */ads_count? As google documentation for SEO says, if you want to block all the bots, it's like u did:

User-agent: * // (to whom? (*) means all bots!
Disallow: /ads_count

Note that directives are case-sensitive. For instance, Disallow: /junk_file.asp would block http://www.example.com/junk_file.asp, but would allow http://www.example.com/Junk_file.asp. Googlebot will ignore white-space (in particular empty lines)and unknown directives in the robots.txt.

answered Dec 04 '12 at 10:43

periback2

1,459
4
19
36

Ex for my url: http://www.abc.com/ads_count/?u=213&a=9 So i try to disallow all bots follow all URLs */ads_count/* Am I doing something wrong? – Vô Danh Vô Hình Dec 05 '12 at 03:23
try adding User-agent: * Disallow: /ads_count*/ (it's supposed to block all sub directories from ads_count – periback2 Dec 05 '12 at 09:43
take a look at this: http://support.google.com/webmasters/bin/answer.py?hl=en&hlrm=pt-BR&answer=156449 – periback2 Dec 05 '12 at 09:44
usually it was supposed to be ok, the way u did. How ru testing it? – periback2 Dec 05 '12 at 09:45

score 0 · Answer 2 · answered Dec 10 '12 at 14:45

Allow and the wildcard * in Disallow are not part of the original robots.txt specification, so not all robots.txt parsers will know/note those rules.

If you want to block all pages starting with /ads_count/, you just need:

User-agent: *
Disallow: /ads_count/

However: not all bots respect robots.txt. So you'd still get hits by bad bots that ignore robots.txt.

How to stop all search engines, bots to crawl some urls

2 Answers2