6

Is there a method where you can only allow robots such as Google, Yahoo, or other search engine robots to my sitemap which is located at http://www.mywebsite.com/sitemap.xml. Is this possible to not allow direct access by a user but only to robots?

MacMac
  • 34,294
  • 55
  • 151
  • 222

3 Answers3

5

Well basically no, but you could do something with the user-agent string and disallow access (assuming Apache)

<Location /sitemap.xml>
  SetEnvIf User-Agent GodBot GoAway=1
  Order allow,deny
  Allow from all
  Deny from env=!GoAway
</Location>

But as it says here (where I found the syntax)

Warning:

Access control by User-Agent is an unreliable technique, since the User-Agent header can be set to anything at all, at the whim of the end user.

Mr Shark
  • 26,068
  • 5
  • 29
  • 37
2

It is in Red in my source:

$ip = $_SERVER["REMOTE_PORT"];
$host = gethostbyaddr($ip);

if(strpos($host, ".googlebot.com") !== false){
    readfile("sitemap.xml");
}else{
    header("Location: /");

  • 1
    This doesn't necessarily help the OP, since he/she didn't indicate a specific language. – Derek Oct 02 '13 at 19:40
0

sitemap.php

<?php

    $ip = $_SERVER["REMOTE_PORT"];
    $host = gethostbyaddr($ip);

    if(strpos($host, ".googlebot.com") !== false){
        readfile("sitemap.xml");
    }else{
        header("Location: /");
    }
Dejan Marjanović
  • 19,244
  • 7
  • 52
  • 66