Extract bot name from user-agent string-Python

Question

I want to extract the botname with its version from user-agent strings. I tried using split function. But since the way of displaying user-agent string is different from one crawler to the other what is the best way to get my expected out put?(Please consider that i need a general solution)

Input(user-agent strings)

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)

Expected output

Googlebot/2.1
AhrefsBot/4.0
msnbot/2.0b

Take a look: http://stackoverflow.com/questions/927552/parsing-http-user-agent-string — alecxe, Sep 03 '13 at 08:52

score 3 · Accepted Answer · answered Sep 03 '13 at 08:50

Try following:

import re

lines = [
    'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
    'Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)',
    'msnbot/2.0b (+http://search.msn.com/msnbot.htm)'
]

botname = re.compile('\w+bot/[.\w]+', flags=re.IGNORECASE)
for line in lines:
    matched = botname.search(line)
    if matched:
        print(matched.group())

prints

Googlebot/2.1
AhrefsBot/4.0
msnbot/2.0b

assumed that bot agent names contain bot/.

Thanks falsetru!...:)This is what I expected! – Nilani Algiriyage Sep 03 '13 at 08:53 — Nilani Algiriyage, Sep 03 '13 at 08:53

Extract bot name from user-agent string-Python

1 Answers1