3

I have 2 web sites. First one is "test-www.xxxxxxx.net" and another one is "www.xxxxx.net". I want to make a dynamic no-index meta tag. Google robots can index my live site but I dont want it for my test site. Normaly I may add an attribute and make it dynamic in _Layout.cshtml as below.

@if (!Helper.IsLiveSite())
    {
        <meta name="robots" content="noindex">
    }

Is there a way that I can make it in robot.txt file ? It is written in some articles that "I strongly recommend you use Noindex instead, whenever possible."

So please help! How can I do it in robot.txt. I dont know if the below file will create an error or not.

User-agent: *
Disallow: /styles/
Sitemap: http://xxxxxx/sitemap/sitemap.xml
Noindex: test-www.xxxxxxx.net/*
Noindex: http://test-www.xxxxxxx.net/*
Noindex: https://test-www.xxxxxxx.net/*

Thanks.

doganilker
  • 103
  • 2
  • 16
  • Google has a robots.txt tester tool: https://www.google.com/webmasters/tools/robots-testing-tool, but basically it's easiest to make a robots.txt file for each of your sites and not one combined for both. Then make the one for your test site `noindex` everything. – Allan S. Hansen Jan 15 '16 at 12:37

1 Answers1

5

You can’t disallow indexing with robots.txt¹, only crawling (with Disallow).

If you want to disallow crawling of all documents from your test site, you have to upload a robots.txt that is accessible from test-www.xxxxxxx.net/robots.txt:

User-agent: *
Disallow: /

(And this robots.txt file should not also be accessible from your live site.)

Search engines (those that support robots.txt) won’t visit (crawl) URLs on this host anymore. If they find URLs to documents on this host somehow (e.g., if another page links to them), they may list (index) them.

¹ Google supported (maybe still supports) this experimentally, though.

unor
  • 92,415
  • 26
  • 211
  • 360
  • Thanks for your help @unor – doganilker Jan 18 '16 at 06:13
  • @unor I have followed your answer. But google still shows my site in search result. I have also put meta tags in my base.html.twig (I am Using symfony2.3). And It shows those metas in source. But Google still shows it in search result. Is there anything I am missing out??? – Ashish Shah Oct 03 '16 at 05:23
  • @AshishShah: How long did you wait after adding this robots.txt? -- Do the results from your site show a description, or does it say that a description can’t be displayed? – unor Oct 03 '16 at 14:17
  • @unor: It has been 2 days after addmin robots.txt. Yes, result shows description. – Ashish Shah Oct 05 '16 at 09:06
  • @AshishShah: You have to wait until the Googlebot visits your site again, which can take way longer than a few days. – unor Oct 05 '16 at 10:30
  • @unor: ok... actually I tested my robots.txt in robits.txt checker provided by google, it has accepted my changes. Still I will wait few days to see what happens. – Ashish Shah Oct 06 '16 at 07:28