59

I want one page of my site to be crawled and no others.

Also, if it's any different than the answer above, I would also like to know the syntax for disallowing everything but the root (index) of the website is.

# robots.txt for http://example.com/

User-agent: *
Disallow: /style-guide
Disallow: /splash
Disallow: /etc
Disallow: /etc
Disallow: /etc
Disallow: /etc
Disallow: /etc

Or can I do like this?

# robots.txt for http://example.com/

User-agent: *
Disallow: /
Allow: /under-construction

Also I should mention that this is a WordPress install, so "under-construction," for example, is set to the front page. So in that case it acts as the index.

I think what I need is to have http://example.com craweld, but no other pages. 

# robots.txt for http://example.com/

User-agent: *
Disallow: /*

Would this mean disallow anything after the root?

unor
  • 92,415
  • 26
  • 211
  • 360
nouveau
  • 1,162
  • 1
  • 8
  • 14

4 Answers4

110

The easiest way to allow access to just one page would be:

User-agent: *
Allow: /under-construction
Disallow: /

The original robots.txt specification says that crawlers should read robots.txt from top to bottom, and use the first matching rule. If you put the Disallow first, then many bots will see it as saying they can't crawl anything. By putting the Allow first, those that apply the rules from top to bottom will see that they can access that page.

The expression rules are simple: the expression Disallow: / says "disallow anything that starts with a slash." So that means everything on the site.

Your Disallow: /* means the same thing to Googlebot and Bingbot, but bots that don't support wildcards could see the /* and think that you meant a literal *. So they could assume that it was okay to crawl /*foo/bar.html.

If you just want to crawl http://example.com, but nothing else, you might try:

Allow: /$
Disallow: /

The $ means "end of string," just like in regular expressions. Again, that'll work for Google and Bing, but won't work for other crawlers if they don't support wildcards.

unor
  • 92,415
  • 26
  • 211
  • 360
Jim Mischel
  • 131,090
  • 20
  • 188
  • 351
  • Jim, you say "end of string", but I believe any url is a string for e.g `http://example.com/new-page.html` or `http://example.com/index.php?route=product/category&path=101`. Pls clarify. – dkjain May 06 '17 at 14:33
  • as suggested by the answer, setting my rotobs.txt to "User-agent: * Allow: /$ Disallow: /" completely removed my website from google search, though I was expecting my root domain to show up (when searching in google with "site:https://example.com") while all other pages/resources from the domain to disappear from google search. What did I miss here? – vir us Feb 14 '18 at 11:33
  • @virus I don't know what the problem is. I suggest you look on Google's site, or contact Google support. – Jim Mischel Feb 14 '18 at 14:54
  • @JimMischel thanks - will try to figure out. Everyone seems to suggest the same solution as you've described in the answer but for whatever reason it doesn't work in my case. weird. Maybe it's related to wordpress somehow? – vir us Feb 14 '18 at 16:38
12

If you log into Google Webmaster Tools, from the left panel go to crawling, then go to Fetch as Google. Here you can test how Google will crawl each page.

In the case of blocking everything but the homepage:

User-agent: *
Allow: /$
Disallow: /

will work.

Kohjah Breese
  • 4,008
  • 6
  • 32
  • 48
  • 12
    How can this block robots.txt if Googlebot needs to view robots.txt to know what to block? –  May 18 '18 at 09:27
4

you can use this below both will work

User-agent: *
Allow: /$
Disallow: /

or

User-agent: *
Allow: /index.php
Disallow: /

the Allow must be before the Disallow because the file is read from top to bottom

Disallow: / says "disallow anything that starts with a slash." So that means everything on the site.

The $ means "end of string," like in regular expressions. so the result of Allow : /$ is your homepage /index

Aominé
  • 472
  • 3
  • 11
2

http://en.wikipedia.org/wiki/Robots.txt#Allow_directive

The order is only important to robots that follow the standard; in the case of the Google or Bing bots, the order is not important.

akx
  • 21
  • 1