0

I couldn't find anything concrete while searching, but while looking at the error logs on my zend server it looks as if web crawlers are modifying various server variables to what seem to be simply inaccurate. It seems to create a sub-domain under my domain, but this sub-domain does not actually exist.

If the actual domain name is www.mydomain.com I get the following values:

'SERVER_NAME' => 'www.arbitraryValue.mydomain.com'
'HTTP_HOST' => 'www.arbitraryValue.mydomain.com'
'SCRIPT_URI' => 'http://www.arbitraryValue.mydomain.com/segment1/segment2/segment3/page.php'

For all of these instances I've noticed the HTTP_USER_AGENT reports these as web crawlers. I've got these incidents from various web crawlers including Google, not just one.

Is this typical behavior of web crawlers? Or should I be concerned about potential Server variable/user agent spoofing?

As far as spoofing I've found this question useful, but if this is typical behavior of web crawlers then I don't have to go down that rabbit hole.

How to fake $_SERVER['REMOTE_ADDR'] variable?

Community
  • 1
  • 1
d.lanza38
  • 2,525
  • 7
  • 30
  • 52
  • So if you go to `www.arbitraryValue.mydomain.com`, does it load in a web browser? I suspect wildcard subdomains. – sjagr Feb 04 '15 at 15:20
  • No, it does not, "Name cannot be resolved". Last I checked there were no wildcard sub-domains and we have no need to make use of them, so a wildcard should not have been added. – d.lanza38 Feb 04 '15 at 15:24
  • It's possible that somewhere there is a link to that subdomain. Web crawlers will follow the link to see if the page exists. In this case there is no spoofing. Spoofing would be that a visitor pretends to be someone else by forging a user agent or even an IP address. In this case it's simply someone visiting an invalid url. The right thing would be to 301 them to the right page (without 'arbitraryValue'), or to return a 404 not found. – GolezTrol Feb 04 '15 at 15:24
  • I'm double checking the use of Wild card sub-domains. I'll re-post once I verify. – d.lanza38 Feb 04 '15 at 17:42

0 Answers0