The x-robots tag is a field in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.
The x-robots tag is a field in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.
A typical x-robots tag:
X-Robots-Tag: noindex
Similar to the meta robots tag, the x-robots tag can contain multiple values, comma-separated.
Below are the most important values:
- all: The default value. No restrictions for robots.
- noindex: Exclude this URL from being indexed
- nofollow: Do not follow links on this page
- none: Do nothing with this page (same as "noindex, nofollow")
The x-robots tag is relatively new and not as widely used or accepted as robots.txt or the meta robots tag. Google, Bing, and Yahoo support it, but other search engines may not.
