0

For example, I would like to block this URL:

example.com/?this-url

Usually I used robots.txt file:

User: *
Disallow: /?$
unor
  • 92,415
  • 26
  • 211
  • 360
johnhgaspay
  • 51
  • 2
  • 10

1 Answers1

1

To ask bots not to index your documents, you have two options: either provide an HTML element or send a HTTP header. The latter option can be used within .htaccess:

Header set X-Robots-Tag "noindex"

(Header directive from the Apache module mod_headers)

unor
  • 92,415
  • 26
  • 211
  • 360