regex: find a string, then look behind

Question

I'm new to regex, so I hope this isn't too obvious a question

I'm looking for the neighborhood in craigslist apartment listing's html. The neighborhood is listed like this

(castro / upper market)
</h2>

And here is an example of the html...

<a class="backup" disabled="disabled">&#9650;</a>
<a class="next" disabled="disabled"> next &#9654;</a>
</span>

</section>

<h2 class="postingtitle">
<span class="star"></span>
&#x0024;5224 / 2br - Stunning Furnished 2BR with Hardwwod Floors &amp; Newly  renovated Kitchen (pacific heights)
</h2>
<section class="userbody">
<figure class="iw">


<div class="slidernav">
    <button class="sliderback">&lt;</button>
    <span class="sliderinfo"></span>
    <button class="sliderforward">&gt;</button>

This should find all the different neighborhoods

But it takes way too long on a full page of html

\w+\s?(\/)?\s?\w+\s?(\/)?\s?\w+\s?(\/)?\s?\w+\)\n<\/h2>

# \w+ to find the word 
# \s?(\/)?\s? for a space or space, forward slash, space
# \n<\/h2> because </h2> is uniquely next to the neighborhood in the html

Is there a way to find

</h2>

Then look behind for the neighborhood string of text?

Thanks so much for any help or steering me in the right direction

Answer 1

Use an HTML Parser to extract the title (h2 tag contents) and then use regular expressions to extract the neighborhood (text inside the parenthesis).

Example (using BeautifulSoup HTML parser):

import re
from bs4 import BeautifulSoup
import requests

response = requests.get('http://sfbay.craigslist.org/sfc/apa/4849806764.html')
soup = BeautifulSoup(response.content)

pattern = re.compile(r'\((.*?)\)$')
text = soup.find('h2', class_='postingtitle').text.strip()
print pattern.search(text).group(1)

Prints pacific heights.

Note the \((.*?)\)$ regular expression - it would capture everything inside the parenthesis that is directly before the end of the string.

---

With Scrapy web-scraping framework you can solve it in one line since Selectors have built-in support for regular expressions. Example from the "Scrapy shell":

$ scrapy shell http://sfbay.craigslist.org/sfc/apa/4849806764.html
In [1]: response.xpath('//h2[@class="postingtitle"]/text()').re(r'\((.*?)\)$')[0]
Out[1]: u'pacific heights'

---

Also see hundred reasons why regex should not be used for HTML parsing:

• RegEx match open tags except XHTML self-contained tags

Answer 2

What about using string.find to find the regex index and then going back negative value at that index.

 In [1]: import re

 In [2]: c = "123456</h2>7890"

 In [3]: x = c.find("</h2>")

 In [4]: print c[x-6:x]
 123456

Answer 3

Assuming your HTML is stored in a variable called page, how about this pattern?

re.findall("\(([^\(\)]+)\)\n<\/h2>", page)

For good measure, allow for extra spaces, too:

re.findall("\(([^\(\)]+)\)\s*\n\s*<\/h2>", page)

Finally, precompile the automaton:

neighborhoods = re.compile( "\(([^\(\)]+)\)\s*\n\s*<\/h2>")

# somewhere else, for each page 
for nh in neighborhoods.findall(page):
    print(nh)

For your example HTML page, this prints the following list of the only neighborhood in there:

pacific heights

If you only have one location per page, re.search() for it would be even faster. Just remember that search() produces an intermediary match object, not the string itself.