Do bots' user agents always have "http" in it?

Question

Is it safe to assume that all bots' user agents always have URLs in the user agent strings? I, of course, compare the user agent against the list of bots, but the idea here to do a preliminary check before check it against a long list.

Perhaps if I could reword my question better, is there any valid non-bot, non-crawler, non-spider or any non-filthy creature that has a URL in the user agent?

@Pekka: I would like to filter out accesses by bots/spiders. — user173457, Nov 02 '10 at 20:55
@user see my related question from a while back http://stackoverflow.com/questions/1717049/tell-bots-apart-from-human-visitors-for-stats — Pekka, Nov 02 '10 at 20:58
@user173457 The Internet doesn't work that way. You put the content up, people have the right to access it how they will. The best you can do is keep a valid [robots.txt](http://www.robotstxt.org/). — user229044, Nov 02 '10 at 21:03

score 2 · Accepted Answer · answered Nov 02 '10 at 20:45

Is it safe to assume that all bots' user agents always have URLs in the user agent strings?

Nope. Check out this bot list, it has plenty of bots that don't sport a URL.

Perhaps if I could reword my question better, is there any valid non-bot, non-crawler, non-spider or any non-filthy creature that has a URL in the user agent?

I can't think of a browser that has a URL in the agent string, but this is definitely a dangerous assumption to work with. Remember that for example, Internet Explorer Add-Ons can add their signatures to the browser's user agent string. You can't guarantee there won't be a URL in it.

score 2 · Answer 2 · edited Oct 07 '21 at 05:59

2

There's no assumptions you can make about the user agent string. RFC 1945, section 10.15 User Agent specifies the format and the section 3.7 Product Token specifies how product tokens should be formatted. As you can see from these two, user agent string can be pretty much anything the HTTP agent wants it to be.

Note: strictly speaking, using an URL in the product token can be treated as a violation of that RFC, since the / should be treated as a separator between the product identifier and the product version.

edited Oct 07 '21 at 05:59

Community

1
1

answered Nov 02 '10 at 20:54

Franci Penov

74,861
18
132
169

+1 for "There's no assumption you can make about the user agent string". – user229044 Nov 02 '10 at 21:01

Do bots' user agents always have "http" in it?

2 Answers2