0

Is noindex an optional directive in a robots.txt file, or are user-agent, disallow, allow and crawl-delay the only options?
For example, is this valid for the contents of a robots.txt file?

user-agent: *  
disallow: /  
noindex: /
Stephen Ostermiller
  • 23,933
  • 14
  • 88
  • 109
dstrube
  • 193
  • 1
  • 6
  • 17
  • See also on webmasters: [How does "Noindex:" in robots.txt work?](https://webmasters.stackexchange.com/questions/84884/how-does-noindex-in-robots-txt-work) In the future non-programming questions about your website should just be asked on [webmasters.se] – Stephen Ostermiller Nov 12 '21 at 10:13

1 Answers1

1

noindex is not a valid directive for a robots.txt file. It is a valid directive for a META robots tag, though.

The only standard directives for robots.txt are "User-agent" and "Disallow". Some browsers support an extended set of directives including "Crawl-delay", "Allow" and "Sitemap". http://rield.com/cheat-sheets/robots-exclusion-standard-protocol seems to have a thorough explanation of the standard and extended directives.

Joe Landsman
  • 2,177
  • 13
  • 9