Stack Overflow
Stack Overflow

Questions tagged [googlebot]

Googlebot is Google's web crawling bot which discovers new and updated pages/documents from the web to build a searchable index for the Google search engine.

Googlebot is the name for Google's web crawler. Depending on the service doing the crawling, Googlebot can present different user agent names.

545 questions
40
votes
9 answers

Is there a way to make search bots ignore certain text?

I have my blog (you can see it if you want, from my profile), and it's fresh, as well as google robots parsing results are. The results were alarming to me. Apparently the most common 2 words on my site are "rss" and "feed", because I use text for…
Alex
  • 14,338
  • 5
  • 41
  • 59
29
votes
2 answers

What does this this HTTP Authorization RewriteRule do?

I have an rewrite recursion error somewhere on my website that Google Bot caused, but I can't find the url that caused it because my Loglevel is low. I raised it but it has not happened again so far. RewriteRule .* -…
The Surrican
  • 29,118
  • 24
  • 122
  • 168
26
votes
5 answers

tag for Google

I would like to tell Google not to index certain parts of the page. In Yandex (russian SE) there's a very useful tag called . How can it be done with Google?
teslasimus
  • 1,238
  • 5
  • 15
  • 23
23
votes
5 answers

How to set up a robot.txt which only allows the default page of a site

Say I have a site on http://example.com. I would really like allowing bots to see the home page, but any other page need to blocked as it is pointless to spider. In other words http://example.com & http://example.com/ should be allowed, but…
Boaz
  • 25,331
  • 21
  • 69
  • 77
23
votes
3 answers

Avoid crawling part of a page with "googleoff" and "googleon"

I am trying to tell Google and other search engines not to crawl some parts of my web page. What I do is: