8

Where should put robots.txt?

domainname.com/robots.txt

or

domainname/public_html/robots.txt

I placed the file in domainname.com/robots.txt, but it's not opening when I type this in browser.

alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png

Tot Zam
  • 8,406
  • 10
  • 51
  • 76
Jitendra Vyas
  • 148,487
  • 229
  • 573
  • 852

4 Answers4

14

Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that.

The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html, for example, put it in domainname/public_html/robots.txt.

VoteyDisciple
  • 37,319
  • 5
  • 97
  • 97
  • thanks it's working now i placed at domainname/public_html/index.html – Jitendra Vyas Jun 06 '10 at 14:55
  • That's not correct. It has to be in the root directory of the web server, which is different if your home page is in a subdirectory. – Mechanical snail Jun 04 '12 at 22:08
  • By "homepage" here I mean the proper homepage for the domain. Some robots will also read `robots.txt` files from subdirectories, but that's not as reliable as "the" robots.txt at a website's root. – VoteyDisciple Jun 05 '12 at 18:22
  • I followed your answer but I cannot access it yet. My question is here http://stackoverflow.com/questions/32302465/i-cannot-access-robots-txt-in-spring-mvc – Jack Aug 31 '15 at 00:46
2

i think the better way to describe it is to have it in the root web folder of your domain... so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.

dont forget: you can use Google Webmaster Tools to check to make sure you haven't restricted anything you didnt mean to(you also get to see queries and links woohoo!).

suggestion: id consider using the <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> if possible because you will still earn linkjuice for links on the page but it wont show up in googles index while a robots.txt directive can leave a plain url with do description in SERPs but will loose all value of links pointed to it because its robots.txted out (its ranking b/c of anchor text so get credit for it)

Carter Cole
  • 918
  • 9
  • 16
2

In the root of your web directory (where you put the files that show up on your website)

Taylor Satula
  • 520
  • 1
  • 5
  • 20
0

In this case you should put it in domainname/public_html/robots.txt, as the public.html folder is where your index file will be.

Sir Graystar
  • 703
  • 1
  • 7
  • 17