8

What are the differences in the lists below:

<meta name="robots" content="index, follow">

<meta name="robots" content="follow">

<meta name="robots" content="noindex, follow">

<meta name="robots" content="index, nofollow">

<meta name="robots" content="noindex, nofollow">
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Richard Frey Reyes
  • 103
  • 1
  • 2
  • 7
  • 2
    It is spelled out in the documentation (e.g. https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta#attr-name , under `robots`.) What exactly do you find unclear about it? – Amadan Jul 11 '18 at 04:51

3 Answers3

11

First of all, you may know about meta robots tags.

Meta robots tags are to tell the spider or crawler which page to crawl or index or not.

Follow means: page will be crawled.

No Follow means: page will not be crawled.

Index means: your page show in search Results.

No Index means: your page does not show in search Results.

<meta name="robots" content="index, follow">

This tag tells the web crawler all pages will be indexed and to crawl them.

<meta name="robots" content="follow">

This tag is used to tell the web crawler all pages will be crawled and indexed.

<meta name="robots" content="noindex, follow">

This tag is used to tell the crawler to follow but not index the page in your database.

<meta name="robots" content="index, nofollow">

To tell the web crawler to not crawl the page, but index them.

<meta name="robots" content="noindex, nofollow">

To tell the web crawler to neither index nor crawl.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Mahesh Sharma
  • 119
  • 7
  • "follow" has nothing to do with the current page getting crawled. The crawler has to crawl the page to see the tag. It has to do with whether the crawler should follow the links on the page to crawl other pages. – Stephen Ostermiller Aug 25 '23 at 19:27
4

The spider will index your whole website. The spider will not only index the first web-page of your website but also all your other web-pages as it crawls through links from this page.

Tells the search engines that it may follow links on the page, to discover other pages (both of these are default behaviour).

<meta name="robots" content="index, follow">

<meta name="robots" content="follow">

The spider will now index your whole website. The spider will not index the web-page but it may follow links on the page, to discover other pages.

<meta name="robots" content="noindex, follow">

The spider will index this page but it won't follow links on this page to new pages.

<meta name="robots" content="index, nofollow">

The spider won't index this page at all and won't follow links on this page to any other pages.

<meta name="robots" content="noindex, nofollow">

Sources:

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Jordan Quartermain
  • 652
  • 3
  • 12
1

The main difference between these tags is, in very simple words:

noindex The page is supposed to not be shown in the result list of pages like google, yahoo and so on. index means the opposite and allows it.

nofollow Disallows crawlers/robots to call (follow) links that are embedded in / found on pages that carry this flag in their meta data. follow means the opposite and allows it.

Now you can figure out yourself what each combination of both results in. ;)