Divide and scrape groups separated by unique using beautiful

Question

So the webpage that I'm trying to scrape is something like this:

...
<tr><td colspan=3><BR><div class="list">Foo:</div></td></tr>
<tr><td><img src="/images/2.gif" alt="Main"> <a href="/foo/1/"></A></td><td><a href="/browse/foo/1/">foo1</A></td></tr>
<tr><td><img src="/images/2.gif" alt="Main"> <a href="/foo/2/"></A></td><td><a href="/browse/foo/2/">foo2</A></td></tr>
<tr><td><img src="/images/1.gif" alt="Guest"> <a href="/foo/3/"></A></td><td><a href="/browse/foo/3/">foo3</A></td></tr>

<tr><td colspan=3><BR><div class="list">Bar:</div></td></tr>
<tr><td><img src="/images/1.gif" alt="Guest"> <a href="/bar/1/"></A></td><td><a href="/browse/bar/1/">bar1</A></td></tr>
<tr><td><img src="/images/1.gif" alt="Guest"> <a href="/bar/2/"></A></td><td><a href="/browse/bar/2/">bar2</A></td></tr>
<tr><td><img src="/images/2.gif" alt="Main"> <a href="/bar/3/"></A></td><td><a href="/browse/bar/3/">bar3</A></td></tr>

<tr><td colspan=3>...

And I would like to scrape data as following:

...
Foo:
    foo1
    foo2
    foo3

Bar:
    bar1
    bar2
    bar3
...

Each contents are separated with <tr> with inscribed <td colspan=3> which make it difficult for me to scrape information... Though I have tried this method I was unable to obtain data since all groups share common <tr> tag.

Would there be rational way to divide those sections using beautifulsoup? Thanks in advance.

I'm not clear on how the method you link to didn't work. Could you show the code that you used and explain what happened? — David Robinson, Apr 15 '13 at 17:58

score 0 · Answer 1 · answered Apr 15 '13 at 18:09

actually I had to deal with a lot of such datastructures lately - td with some span is a header and next tds are contents until new td with span comes

I use something like this.

class TrGrouper:
    def __init__(self, attr='colspan'):
        self.value = 0 
        self.attr = attr

    def __call__(self, tr):
        first_cell = tr.th if tr.th else tr.td
        self.value += int(first_cell.get(self.attr))
        return self.value


#usage:

import itertools

list_of_TRs = soup.find_all('tr')

for _, chunk in itertools.groupby(list_of_TRs, TrGrouper()):
    # do_something 
    # note that chunk is a generator

#or

grouped_rows = [list(chunk) for _, chunk in itertools.groupby(list_of_TRs, TrGrouper())]

Divide and scrape groups separated by unique using beautiful

1 Answers1