Adding a HTML element to the robots.txt

Question

I am constructing a website for educational purposes. I know robots.txt can be used to allow or disallow user-agents, queries and directory access. Now I have a canvas element in one page that needs to be activated only is specific user agent is used to access that page. Else it should throw an error message. Is this possible?

score 3 · Accepted Answer · answered Jul 11 '15 at 21:57

3

You'll have to do this with Javascript.

You can get the user agent of a browser via:

window.navigator.userAgent

You can then choose to display or not display the canvas element accordingly. You won't have access to your robots.txt directly though, unless you include it in your in JS.

Documentation

answered Jul 11 '15 at 21:57

Breedly

12,838
13
59
83

Thank you for the fast reply and pointing that fact. Really appreciated :) – Ashwin Asp Jul 12 '15 at 08:45

score 0 · Answer 2 · answered Jul 11 '15 at 22:11

The robots.txt file is intended for search engines and other web crawlers, not for normal users. It sounds like you want to target the latter, which makes robots.txt unsuitable for this task. (Besides: it's not possible, even if you were targeting web crawlers).

You'll have to use a different way to "sniff" the user's browser type. JavaScript is certainly a good possiblity. If you intend to do more complex things with scripts executed on the server (PHP or the like), you could use those languages to provide different contents to different browsers. The web is full of tutorials which may be helpful to you.

dakab · Answer 3 · 2015-07-12T18:17:14.270

0

You already described the extend of what a robots.txt is for. Its list of links must be ignored and not be indexed by webcrawlers, this is its purpose. On a web page however, you can use JavaScript to determine the user-agent with the navigator.userAgent property.

Depending on your implementation, you can hide the <canvas> element and check for your user-agent at runtime with a simplistic logic like this:

if (window.navigator.userAgent == 'SpecificUserAgent/32')
    $("#special-canvas").show();
else
    alert("Please come back later with another user-agent.");

Now, the only question remains is what your “specific user agent” actually looks like. Typical browser user agents comprise a complex build-up:

    Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/39.0
    Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16

So, if you’d like to show your canvas only to the latter, you probably need a regular expression:

if(/Chrome\/43/.test(window.navigator.userAgent))
    alert("only if user agents contains 'Chrome/43'.");

Finally, there are addons like User Agent Switcher for Firefox and other browsers that let you switch your user agent. Your browser’s current one can easily be seen at sites that display your UA or using alert(window.navigator.userAgent);.

edited Jul 12 '15 at 18:17

answered Jul 11 '15 at 22:44

dakab

5,379
9
43
67

I tried this. I installed User-Agent Switcher for google chrome. Tried custom and given user-agents. When I use this code I always get "Unknown browser". function test() { if(window.navigator.userAgent == "iPad") { alert('iPad Detected'); } else { alert('Unknown Browser'); } } Am i missing something? – Ashwin Asp Jul 12 '15 at 08:43
The user-agent of an iPad is far more complex (see [this answer](http://stackoverflow.com/a/2248420/2083613) for instance), it won’t be just “iPad”. An immediate fix would be `if(window.navigator.userAgent.indexOf('iPad') > -1)`. – dakab Jul 12 '15 at 09:28
iPad is just an example i tried opera, ie, firefox, custom always returns false – Ashwin Asp Jul 12 '15 at 16:33
These were only examples, mere placeholders. See [Wikipedia for format examples](https://en.wikipedia.org/wiki/User_agent#Format_for_human-operated_web_browsers). If you tell us your “specific user agent”, we can provide a solution for that. – dakab Jul 12 '15 at 17:59
The same example works on Firefox with "User Agent Switcher" Addon. Further debugging shows that in chrome the user agent is not actually changing. It stays the default user agent string "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36". Can anyone tell me why? – Ashwin Asp Jul 12 '15 at 18:06
@AshwinAsp: Please take a look at my edited answer in response to your additional questions. – dakab Jul 13 '15 at 11:39

Adding a HTML element to the robots.txt

3 Answers3