fast parsing links out of a page in python

Question

I need to parse a large number of pages (say 1000) and replace the links with tinyurl links.

right now i am doing this using a regex

href_link_re = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)

but its not fast enough.

i am thinking so far

state machine (the success of this will depend on my ability to write clever code)
using an html parser

Can you suggest faster ways?

EDIT: You would think that an html parser would be faster than regex, but in my tests it is not:

from BeautifulSoup import BeautifulSoup, SoupStrainer

import re
import time

__author__ = 'misha'

regex = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)

def test(text, fn, desc):

    start = time.time()
    total  = 0
    links = [];
    for i in range(0, 10):
        links = fn(text)
        total += len(links)
    end = time.time()
    print(desc % (end-start, total))
   # print(links)

def parseRegex(text):
    links  = set([])
    for link in regex.findall(text):
        links.add(link[1])
    return links

def parseSoup(text):
    links = set([])
    for link in BeautifulSoup(text, parseOnlyThese=SoupStrainer('a')):
        if link.has_key('href'):
            links.add(link['href'])

    return links



if __name__ == '__main__':
    f = open('/Users/misha/test')
    text = ''.join(f.readlines())
    f.close()

    test(text, parseRegex, "regex time taken: %s found links: %s" )
    test(text, parseSoup, "soup time taken: %s found links: %s" )

output:

regex time taken: 0.00451803207397 found links: 2450
soup time taken: 0.791836977005 found links: 2450

(test is a dump of the wikipedia front page)

i must be using soup badly. what am i doing wrong?

Why would an HTML parser be faster than regex? Besides, BeautifulSoup is largely Python code, whereas the re module is C. — susmits, Jun 03 '11 at 15:06

score 2 · Accepted Answer · answered Jun 04 '11 at 16:40

2

LXML is probably your best bet for this task. See Beautiful Soup vs LXML Performance. Parsing links is easy in LXML and it's fast.

root = lxml.html.fromstring(s)
anchors = root.cssselect("a")
links = [a.get("href") for a in anchors]

answered Jun 04 '11 at 16:40

jfocht

1,704
1
12
16

score 1 · Answer 2 · answered Jun 02 '11 at 20:50

1

Parsing using regexp's very bad idea, because of speed and regexp exponential time problem. Instead you can use parsers for xhtml. The best is LXML. Or you can write parser specially for this purpose with LL,LR parsers. For example: ANTLR,YAPPS,YACC,PYBISON, etc

answered Jun 02 '11 at 20:50

Andrey Nikishaev

3,759
5
40
55

2

I remember this legendary SO post every time a question is posted involving using regexes to parse html. http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags – Aphex Jun 02 '11 at 20:56
Which is faster: LXML or Beautiful Soup? I've been using the latter, but it seems like I see the former being recommended more often these days – jhocking Jun 02 '11 at 20:56
LXML is native parser, while Beautiful Soup use reg exps for parsing. So LXML is better. – Andrey Nikishaev Jun 05 '11 at 19:58

fast parsing links out of a page in python

2 Answers2