3

I found the following question but if possible I don't want to have to make changes that will affect all sites on the server. HttpBrowserCapabilities.Crawler property .NET

I also found instructions on using the browserCaps Element http://msdn.microsoft.com/en-us/library/sk9az15a.aspx but it seems to be deprecated.

What is the easiest way to detect the major search engines in Asp.net 4.0?

(I am only really interested in being able to detect major search engines)

Community
  • 1
  • 1
Tom
  • 12,591
  • 13
  • 72
  • 112
  • 2
    Do you want the crawler or the referrer? – Daniel A. White Dec 01 '11 at 18:07
  • @DanielA.White I have a site with an age verification screen which legally has to be shown to users, but if I don't properly detect search engines this is getting indexed so I need to be able to detect search engines – Tom Dec 02 '11 at 17:00
  • Not MVC, but an option for you here: http://stackoverflow.com/questions/544450/detecting-honest-web-crawlers – Dave Sumter Jan 26 '13 at 11:28

2 Answers2

0

Just the same way as you would in a normal ASP.Net site.

        if( Request.Browser.Crawler )
        {

        }

You can add an httpmodule to do detection on all requests.

Dave Walker
  • 3,498
  • 1
  • 24
  • 25
  • Yep I did try this but google wasn't being detected, the link I included in my question suggested that the browser list to be updated but the solution seemed to be deprecated since it was asked 3 years ago – Tom Dec 02 '11 at 17:05
0

Checkout Browser Caps here

Or

Get the HttpRequest.UserAgent and check it against a list of know user agents.

Tr1stan
  • 2,755
  • 1
  • 26
  • 45
  • He does not seem to be wanting to *hide* content from Search Engines; he wants to explicitly *show* content to them, whereas normal users actually *can't* see that content until they verify their age. So, robots.txt would not be helpful. Nor would that list of search engine IPs; you should never try to use something so brittle, IMO. – Andrew Barber Jan 23 '12 at 08:34
  • @AndrewBarber: You're absolutely right, I misread his comment (well I interprated it wrong). Answer adjusted. – Tr1stan Jan 23 '12 at 08:51