How to disallow all for all user agents except one user agent? For example disallow all for all user agent, but allow for Googlebot only?
Asked
Active
Viewed 8,478 times
0
-
1This question is somewhat of a duplicate: http://stackoverflow.com/questions/671491/robots-txt-allow-only-major-se – CristianHG Nov 09 '15 at 20:48
1 Answers
1
User-agent: *
Disallow: /
User-agent: google
Allow: /
This sample robots.txt tells crawlers that if they are not with google. then it is preferred they don't crawl your site. While google has been given the greenpass to crawl anything on the site.
This file should be stored at www.example.com/robots.txt.
Please read up on robots.txt
user1256378
- 712
- 2
- 12
- 31
Kent Kostelac
- 2,257
- 3
- 29
- 42
-
It's the first time I see `=` instead of `:` after `(dis)allow`. Is this syntax correct? – ilyaigpetrov Jun 06 '18 at 12:40
-
Maybe syntax changed since this answer but there is no such syntax "Allow: /". You must use "Disallow:" with no path. – twistedpixel Sep 13 '19 at 11:49