Is there a method where you can only allow robots such as Google, Yahoo, or other search engine robots to my sitemap which is located at http://www.mywebsite.com/sitemap.xml. Is this possible to not allow direct access by a user but only to robots?
Asked
Active
Viewed 1,371 times
3 Answers
5
Well basically no, but you could do something with the user-agent string and disallow access (assuming Apache)
<Location /sitemap.xml>
SetEnvIf User-Agent GodBot GoAway=1
Order allow,deny
Allow from all
Deny from env=!GoAway
</Location>
But as it says here (where I found the syntax)
Warning:
Access control by User-Agent is an unreliable technique, since the User-Agent header can be set to anything at all, at the whim of the end user.

Mr Shark
- 26,068
- 5
- 29
- 37
2
It is in Red in my source:
$ip = $_SERVER["REMOTE_PORT"];
$host = gethostbyaddr($ip);
if(strpos($host, ".googlebot.com") !== false){
readfile("sitemap.xml");
}else{
header("Location: /");

Mileen Coulter
- 21
- 1
-
1This doesn't necessarily help the OP, since he/she didn't indicate a specific language. – Derek Oct 02 '13 at 19:40
0
sitemap.php
<?php
$ip = $_SERVER["REMOTE_PORT"];
$host = gethostbyaddr($ip);
if(strpos($host, ".googlebot.com") !== false){
readfile("sitemap.xml");
}else{
header("Location: /");
}

Dejan Marjanović
- 19,244
- 7
- 52
- 66