1

I have a (Rails) site and I want the search engines to crawl and index it. However, I also have some actions that I want to log as having happened - and these actions can be triggered by logged in users as well as users not logged in. Now, to ensure that the count for non-logged in ie anonymous users doesn't include bot traffic I am considering a few options and am looking for guidance on which way to go:

  1. Set a cookie for all users, if this cookie doesn't come back since Bots usually dont accept or send back cookies, I can distinguish bots from anonymous humans.

  2. Check the header and see if the agent is a bot (some whitelist): How to recognize bots with php?

  3. Set that action to be a POST rather than a GET. Bots issue GETs so they don't get counted.

  4. Any other approaches?

I am sure folks have had to do this before so what's the 'canonical' way to solve this?

Community
  • 1
  • 1
bachposer
  • 953
  • 3
  • 8
  • 17

2 Answers2

1

If you don't want the spiders to follow the links, then you can use rel="nofollow" on them. However, since there might be other links pointing into the pages, you will probably also want to look at the User-Agent header. In my experience, the most common User-Agent headers are:

  • Google: Googlebot/2.1 ( http://www.googlebot.com/bot.html)
  • Google Image: Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
  • MSN Live: msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
  • Yahoo: Mozilla/5.0 (compatible; Yahoo! Slurp;)
Martin Geisler
  • 72,968
  • 25
  • 171
  • 229
  • No problem — as you probably know, it's mostly used to prevent manipulating page rank when a site shows user-submitted external links, so your use case is a little different. – Martin Geisler Dec 18 '11 at 09:26
0

Just check the User-Agent header, that might be enough for your purposes. Note that a user agent can just pose as Google bot. So if you want to be sure more checking is needed. But I don't think you'd need to bother further than this.

Lumi
  • 14,775
  • 8
  • 59
  • 92