Stack Overflow
Stack Overflow

Questions tagged [noindex]

The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page.

The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page.

Example:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Don't index this page</title>
</head>
128 questions
26
votes
5 answers

tag for Google

I would like to tell Google not to index certain parts of the page. In Yandex (russian SE) there's a very useful tag called . How can it be done with Google?
teslasimus
  • 1,238
  • 5
  • 15
  • 23
16
votes
1 answer

"rel=nofollow noopener" - Possible to have both at the same time?

I've got a pdf that I'd like to protect and don't want search engines to index it. Currently, my link is as follows: View PDF Would I be able to add nofollow tag to the rel…
Joe Bloggs
  • 1,410
  • 2
  • 24
  • 53
14
votes
3 answers

Precedence of X-Robots-Tag header vs robots meta tag

I've placed the following Header in my vhost config: Header set X-Robots-Tag "noindex, nofollow" The goal here is to just disable search engines from indexing my testing environment. The site is Wordpress and there is a plugin installed to manage…
Jared Eitnier
  • 7,012
  • 12
  • 68
  • 123
8
votes
3 answers

What is the difference between "index, follow" or "follow"?

What are the differences in the lists below:
Richard Frey Reyes
  • 103
  • 1
  • 2
  • 7
7
votes
1 answer

FastAPI, robots.txt and noindex

Does FastAPI need robots.txt and the tag noindex? I am creating business api app which shouldn't be called by anonymous. So I wonder whether I have to prepare robots.txt and the tag noindex in order to avoid any crawler's action or not. I made…
tomo
  • 71
  • 2
7
votes
4 answers

X-Robots noindex specific page in .htaccess

Can I 'noindex, follow' a specific page using x robots in .htaccess? I've found some instructions for noindexing types of files, but I can't find instruction to noindex a single page, and what I have tried so far hasn't worked. This is the page I'm…
moobot
  • 477
  • 1
  • 4
  • 20
5
votes
2 answers

Setting noindex on Amazon S3 objects

We have some publicly shared S3 files that we want to make sure won't be indexed by Google. I can't seem to find any documentation on how to do this. Is there a way to set a "noindex" x-robots-tag response header on individual S3 objects? (We're…
Yarin
  • 173,523
  • 149
  • 402
  • 512
4
votes
1 answer

How to set the X-Robots-Tag HTTP header via .htaccess file based on URL query string

Is it possible to apply HTTP header directives based on the URL's query string using an apache .htaccess? For example, based on this resource http://code.google.com/web/controlcrawlindex/docs/robots_meta_tag.html under the section titled "Practical…
John Erck
  • 9,478
  • 8
  • 61
  • 71
4
votes
3 answers

How to tell Jekyll to hide one page from search engines?

I have a website consisting of my public profile, made in Jekyll. It also contains one page, say 'details.html', which contains more personal information about me. I want only those people to see this page whom I give out the link to. In particular,…
Alexander Engelhardt
  • 1,632
  • 3
  • 16
  • 31
4
votes
1 answer

X Robots Tag noindex specific page

I have a Privacy Policy page on my website www.domain/privacy-policy/ which I would like to noindex with the X Robots Tag. I have tried the following code but it does not match # BEGIN WordPress RewriteEngine On RewriteBase…
bondimedical
  • 201
  • 4
  • 11
4
votes
1 answer

How can I check if page has noindex?

My intention is to fetch the link with PHP and maybe with Simple PHP DOM parser (or something similar) parse the content and look for H1-H6 tags. But prior to that I would need to find out if the page is being indexed at all. Other than parsing the…
Ivan Topić
  • 3,064
  • 6
  • 34
  • 47
3
votes
3 answers

Does a 'user'-url based website create issues with Google / Search Engines?

I currently maintain the backend php code for a website which allows our sales representatives to sell products and services. If a sales representative is active, he/she is given a "custom" website URL which essentially tags any activity on that…
JM4
  • 6,740
  • 18
  • 77
  • 125
3
votes
3 answers

Google, do not index YET

In the effort of building a live site on its actual live hosting platform is there a way to tell google to not YET index the website? I found the following: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710 But would that tell…
Scott Rowley
  • 486
  • 1
  • 7
  • 30
3
votes
1 answer

robots.txt: Disallow bots to access a given "url depth"

I have links with this structure: http://www.example.com/tags/blah http://www.example.com/tags/blubb http://www.example.com/tags/blah/blubb (for all items that match BOTH tags) I want google & co to spider all links that have ONE tag in the URL,…
BlaM
  • 28,465
  • 32
  • 91
  • 105
3
votes
2 answers

Noindex in a robots.txt

I've always stopped google from indexing my website using a robots.txt file. Recently i've read an article from a google employee where he stated you should do this using meta tags. Does this mean Robots.txt won't work? Since i'm working with a CMS…
Kevin
  • 83
  • 9
1
2 3
…
8 9