0

Right now I have a situation where my terms of use is loaded through an ascx control and is shown w/ jquery when the user needs or wants to see it... I like this UX because I can show the TOU in a jquery pop up that I control without redirecting the user.

Problem is that google has picked up on this because it's loaded on by the ascx as static text. I know I could put this in a new page and just redirect (and add an ignore rule to the robots.txt), but my question, is if I load this text async w/ ajax only when it needs to be shown, will google's web crawler click on every link and still attribute all this legal text to the page (and continue to screw up my seo results).

Before I try this technique, I wondered if anyone 1) Had any other technique that they use or 2) know if this technique will definitively work or not. Google takes a few days to index my site, so I didn't want to waste time if someone knew for certain.

Please note that the only related question I found was here, but this guy was trying to be nefarious against the web crawlers. I just want the robots.txt ignore behavior without compromising my UX.

Community
  • 1
  • 1
Rikon
  • 2,688
  • 3
  • 22
  • 32

1 Answers1

1

One thing you can do is only display your Terms of Use when the User-Agent (i.e., the user's browser) is not Googlebot.

In ASP.NET, you can verify the browser by checking Request.ServerVariables("HTTP_USER_AGENT").

Google reports the following user-agent:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

So, simply wrap your content with a check, like so:

if (! Request.ServerVariables("HTTP_USER_AGENT").Contains("Googlebot"))
{
     <your content here>
}

Good luck!

Roy Dictus
  • 32,551
  • 8
  • 60
  • 76