For example, I would like to block this URL:
example.com/?this-url
Usually I used robots.txt file:
User: *
Disallow: /?$
For example, I would like to block this URL:
example.com/?this-url
Usually I used robots.txt file:
User: *
Disallow: /?$
To ask bots not to index your documents, you have two options: either provide an HTML element or send a HTTP header. The latter option can be used within .htaccess:
Header set X-Robots-Tag "noindex"
(Header
directive from the Apache module mod_headers
)