2

I don't want to see Robots.txt in result list of search. I'm working on an asp.net project.

I found some failed solutions, one of them like below:

  • Using "X-Robots-Tag" in header.

But I want to hide a txt file so there is no "header" part in it.

Thanks for help.

emy
  • 664
  • 1
  • 9
  • 22

2 Answers2

1

The capitalization is important to most popular search engines, including Google (source). Change your file name from Robots.txt to robots.txt. If you name it correctly, most popular search engines (Bing, Yahoo, Google, etc.) will not index it. No extra effort on your part.

Here is Yahoo's crawler help that describes case-sensitivity.

Here is the standard that describes common courtesy for crawlers with regards to robots.txt. It is explicitly stated that empty robots.txt files are ignored completely, and it is common sense that they are not indexed. Otherwise, we would see billions of robots.txt search results in Google.

Rainbolt
  • 3,542
  • 1
  • 20
  • 44
  • Can you give me a source for your first sentence ("Most popular search engines (Bing, Yahoo, Google, etc.) look for robots.txt at the root of your hierarchy. They do not index it. No extra effort on your part."). Because i must prove it to my boss :) . – emy Feb 13 '14 at 15:21
  • @Hayzum Your comment was posted two minutes post-edit. – Rainbolt Feb 13 '14 at 15:24
  • @Hayzum Tell your boss that nothing can stop a web crawler from ignoring common courtesy and indexing ALL of your files. Web crawlers do not have to follow these rules if they choose not to. But why would your boss care about the availability of your robots.txt file? There should be no sensitive information there. – Rainbolt Feb 13 '14 at 15:29
0

A txt file sent via http has a header(containing type:text/plain) just take request and send txt with your own headers.

Eddynand Fuchs
  • 344
  • 1
  • 13