Does related subfolders need to be disallowed separately in robots.txt?

Question

Will disallowing certain folder in robots.txt disallow its related subfolders? Example:

Disallow:/folder/

Will match:

/folder/page
/folder/subfolder/page

Or it will just match:

/folder/page

So if the second case is true, do I need to disallow second and subsequent subfolder separately?

Disallow: /folder/

Disallow  /folder/subfolder/

Disallow /folder/subfolder/onemorefolder

score 0 · Answer 1 · answered Apr 30 '14 at 10:57

0

Robots.txt has no concept of "folders", it’s just strings. Whatever you specify in Disallow is the beginning of the URL path.

Disallow: / blocks any URL whose path starts with / (= all pages).

Disallow: /foo blocks any URL whose path starts with /foo:

Disallow: /foo/ blocks any URL whose path starts with /foo/:

answered Apr 30 '14 at 10:57

unor

Very good explanation. Thanks! I believe the same concept is used in bunch of other similar things, like gitignore file for example. I wonder what other special characters are allowed when making you rules, except asterisk. – Vasyl Nekohda Sep 16 '15 at 12:32
@vaseech: The original robots.txt specification doesn’t define *any* special characters with reserved meanings for `Disallow` values. So if you have `Disallow: /foo*`, the `*` is interpreted literally, i.e., URLs that contain the asterisk. However, specific consumers (like the Googlebot) often extend the robots.txt specification and interpret certain characters differently. – unor Sep 17 '15 at 14:54

1 Answers1