Python: How do you use re to ignore links in parentheses?

Question

The relevant part of the code is:

import re
reargs = '<a\s*href=[\'|"](.*?)[\'"].*?>'
link = re.search(reargs,content,flags=re.IGNORECASE)

I'm building a crawler and the web pages I'm working with have links in parentheses that I don't want so it would be like:

Foo foo foo foo (link) foo foo foo foo link foo foo foo foo (foo link foo) foo foo link foo foo link......and so on

Don't use RegEx to parse HTML. Use an (X)HTML Parser. BeautifulSoup comes to mind. — g.d.d.c, Sep 13 '11 at 18:07
[Parsing HTML/XML with regexes can be problematic.](http://stackoverflow.com/questions/1732348/1732454#1732454) — Tom Zych, Sep 13 '11 at 18:11
To add just a hint more detail, while it is possible with modern regular expressions to do things like you're asking, it generally isn't the best idea. A good clue that you should be using some different technology is the fact that you're trying to match balanced parenthesis and have the possibility for nested items. — Wilduck, Sep 13 '11 at 18:15

score 0 · Answer 1 · answered Sep 13 '11 at 18:30

If there can be multiple sets of nested parentheses like "((foo) link)", I don't think this is possible with regular expressions. In particular, note that parentheses can be used inside URLs (such as at wikipedia), so there may still be nested parens even if the text itself doesn't contain any. So, in the general case I don't think this can be done with regex.

In order to solve it, I will assume you can have parentheses at most 1 level deep, and that no URLs contain parentheses.

The regex you're looking for is something like the following:

(\([^\)]*\)|[^\(<])*_link_

Where _link_ is a regular expression matching a link (which you describe in the problem statement, though it might need some tweaking). To summarize what that first part of my regex is: it matches 0 or more of either a parenthetical statement or a non-link non-parenthesis character. Now, use the matched back references (link.group(2) in your example) to find your URL.

score 0 · Answer 2 · answered Sep 13 '11 at 18:31

In general parsing HTML with regex is a bad idea. But because you asked, and the general question has merit (how to ignore cases where your match is surrounded by parentheses) I'll tell you what I think.

Now, because I don't know what your page looks like I'll just say that, in general, you can exclude matches by adding [^x],except where x is the character you don't want. The brackets make it so that it will match anything, and the ^ excludes whatever follows.

So you can exclude parentheses by surround your match string with [^(]foo[^)]. If there are other characters between the parentheses you'll have to account for that separately.

score 0 · Answer 3 · answered Sep 13 '11 at 18:35

With lxml you could do something like this:

import lxml.html
import re

tree = lxml.html.parse("http://pastehtml.com/view/b7604in99.html")
links = tree.xpath("//a")

for link in links:
    if re.match(r'^\(.*\)$', link.text.strip()):
        print link.get('href')

Python: How do you use re to ignore links in parentheses?

3 Answers3