3

I have links with this structure:

I want google & co to spider all links that have ONE tag in the URL, but NOT the URLs that have two or more tags.

Currently I use the html meta tag "robots" -> "noindex, nofollow" to solve the problem.

Is there a robots.txt solution (that works at least for some search bots) or do I need to continue with "noindex, nofollow" and live with the additional traffic?

Community
  • 1
  • 1
BlaM
  • 28,465
  • 32
  • 91
  • 105

1 Answers1

1

I don't think you can do it using robots.txt. The standard is pretty narrow (no wildcards, must be at the top level, etc.).

What about disallowing them based on user-agent in your server?

MarkusQ
  • 21,814
  • 3
  • 56
  • 68
  • Disallowing access would result in some kind of HTTP error. Not sure how google reacts to pages with lots of "server error". Not very enthusiastic to try out :) – BlaM Mar 25 '09 at 18:39
  • Wouldn't have to--you could serve up some cheap static "nothing to see here" page. – MarkusQ Mar 25 '09 at 18:44
  • That's true. Would at least be better than to serve the full page. – BlaM Mar 25 '09 at 18:47