5

I'm using an action filter that checks what browser version is being used on my site, if it's an older browser I put up a div at the top asking them to upgrade. I don't want web crawlers to get the div message, so I've implemented HttpBrowserCapabilitiesBase.Crawler and it looks like it works for Google, but Bing and the others don't seem to register as crawlers. Strange for a Microsoft product to not notice Bing as a crawler!

Is there some way to add user agents to the crawler property or something?

Thanks!

Edited: I'm using asp.net mvc 3, it looks like I need to user .Browser files(?). Anyone know of a comprehensive set of .Browser files out there for Bing and the rest of the crawlers?

tereško
  • 58,060
  • 25
  • 98
  • 150
gt124
  • 1,238
  • 13
  • 23

2 Answers2

5

You will probably need to update your browscap.ini file as the one shipped with IIS is probably old. You can get a new one at one of the following URLs:

browscap.ini usually lives at: c:\windows\system32\inetsrv\browscap.ini

Deleted
  • 4,804
  • 1
  • 22
  • 17
  • 1
    This doesn't seem to be working for me, it's probably because I'm using asp.net mvc? Further research was telling me to use .Browser files. – gt124 Sep 29 '11 at 17:36
  • Either should work AFAIK. We're using ASP.Net MVC and it worked for us. Are you sure the crawler is in the browscap file? – Deleted Sep 29 '11 at 19:27
  • I added the browscap.ini from garykeith link you've provided, then I went into the framework/v4/config/browsers directory, edited gateway.browser and added bingbot next to msnbot for the crawler gateway, compiled with framework/v4/aspnet_regbrowsers.exe -i and... It started working. – gt124 Sep 29 '11 at 22:53
2

We're not using MVC but we do this:

Regex.IsMatch(Request.UserAgent, @"bot|crawler|baiduspider|80legs|ia_archiver|voyager|curl|wget|yahoo! slurp|mediapartners-google", RegexOptions.IgnoreCase);

More options in my answer here:

Detecting honest web crawlers

Community
  • 1
  • 1
Dave Sumter
  • 2,926
  • 1
  • 21
  • 29