14

I've placed the following Header in my vhost config:

Header set X-Robots-Tag "noindex, nofollow"

The goal here is to just disable search engines from indexing my testing environment. The site is Wordpress and there is a plugin installed to manage per-page the meta robots settings. For example:

<meta name="robots" content="index, follow" />

So my question is, which directive will take precedence over the other since both are being set on every page?

Yarin
  • 173,523
  • 149
  • 402
  • 512
Jared Eitnier
  • 7,012
  • 12
  • 68
  • 123
  • Great question. My suspicion is that, since `index`,`follow` are the *default* values, any `noindex`,`nofollow` directives that show up, whether in header or meta, will trump it. But I'd like to hear a definitive answer. Adding bounty... – Yarin Aug 16 '13 at 18:04

3 Answers3

19

I am not sure if a definitive answer can be given to the question, as the behavior may be implementation-dependent (on the robot side).

However, I think there is reasonable evidence that X-Robots-Tag will take precedence over <meta name="robots" .... See :

One significant difference between the X-Robots-Tag and the robots meta directive is:

  • X-Robots-Tag is part of the HTTP protocol header.
  • <meta name="robots" ... is part of the HTML document header.

Therefore the the X-Robots-Tag belongs to HTTP protocol layer, while <meta name="robots" ... belongs to the HTML protocol layer.

Protocol capture

As they belong to a different protocol layer, they will not be parsed simultaneously by the (robot) client getting the page: The HTTP layer will be parsed first, and the HTML in a later step.

(Also, it should be noted that X-Robots-Tag and <meta name="robots" ... are not suppported by all robots. Google and Yahoo/Bing suppport both, but according to this some support only <meta name="robots" ..., others support neither.)

Summary :

  • if supported by the robot, X-Robots-Tag will be processed first ; restrictions (noindex, nofollow) apply (and <meta name="robots" ... is ignored).
  • else, <meta name="robots" ... directive applies.
Yarin
  • 173,523
  • 149
  • 402
  • 512
d-stroyer
  • 2,638
  • 2
  • 19
  • 31
  • The way this is explained makes perfect sense to me and I would have to agree with the facts. Can anyone else confirm this? – Jared Eitnier Aug 20 '13 at 13:33
  • In my case (http header contains X-Robots-Tag as "noindex, nofollow" AND html document head contains robots meta tag as "index,follow"), I was expecting that page would be removed from search results. BUT after waiting 3 days it seems doesn't work. Below answer from @Dan may be correct. So I will modify the html head to make it compatible with http header X-Robots-Tag and will observe the results. – csonuryilmaz Mar 20 '17 at 11:29
4

Just an update to Dan's experience, I also have both the 

Header set X-Robots-Tag "noindex, nofollow"

and 

<meta name="robots" content="index, follow" />

on my one of my Wordpress sites, and a check in Google Search Console confirmed that the noindex in X-Robots-Tag is taking precedence as the pages have been crawled and but aren't indexed. So the logic in the correct answer is indeed, correct.

Marie
  • 76
  • 4
2

In my recent experience, when Google sees mixed-messages it prefers positive action by default - ie - it favours indexing - meanwhile will flag the issue as a critical error/warning in your webmaster tools console if you have one.

see your site's status in google here: https://www.google.com/webmasters/

see you site's status in bing here: http://www.bing.com/toolbox/webmaster (note that yahoo search is now powered by bing)

Google takes this positive-by-default action because lots of site owners unwittingly have a dodgy cms semi-blocking robots and we know how google loves to accumulate as much data as it can - any excuse!

if the technical settings are erroneous they're liable to be totally disregarded, and we know how search engines index and follow by default when no settings are specified.

Dan
  • 21
  • 2