How does the HttpBrowserCapabilities.Crawler property (http://msdn.microsoft.com/en-us/library/aa332775(VS.71).aspx) work? I need to detect a partner's custom crawler and this property is returning false.
Where/How can I add his user agent so that this property will return true? Any other way outside of creating my own user agent detecting mechanism?
This is all driven by the default browsercaps declarations that are part of the .NET framework.
To setup this specific crawler, you would need to add a custom browserCaps section to your web.config to identify it as a crawler.
The MSDN documentation on how to go about this is pretty well documented, here is a link.
HttpBrowserCapabilities is populated from a set of *.browser files in C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG\Browsers. You can add your own file in ~/App_Browsers/ with the details of your custom crawler. The schema is here. Basically it contains a regular expression to match against the User-Agent header and a collection of property values.
