How do I extract table data in pairs using BeautifulSoup?

Question

My data sample :

<table id = "history">
<tr class = "printCol">
<td class="name">Google</td><td class="date">07/11/2001</td><td class="state">
<span>CA</span>
</td>
</tr>
<tr class = "printCol">
<td class="name">Apple</td><td class="date">27/08/2001</td>
</tr>
<tr class = "printCol">
<td class="name">Microsoft</td><td class="date">01/11/1991</td>
</tr>
</table>

Beautifulsoup code :

table = soup.find("table", id = "history")

rows = table.findAll('tr')
for tr in rows:
    cols = tr.findAll('td')
    for td in cols:
        print td.find(text=True)

Desired Output for MySQL storage (list):

['Google|07/11/2001|CA', 'Apple|27/08/2001', 'Microsoft|01/11/1991']

Output I have (difficult to associate the right date to the right company) :

Google
07/11/2001


Apple
27/08/2001
Microsoft
01/11/1991

I wrote a function to extract elements from each tr but I thought there is a much more efficient way of doing it all in the original for loop. I want to store them in a list as data pairs. Thoughts?

Sean Vieira · Accepted Answer · 2011-11-15T22:27:11.337

20

List comprehension will make it easier:

table = soup.find("table", id = "history")
rows = table.findAll('tr')
data = [[td.findChildren(text=True) for td in tr.findAll("td")] for tr in rows]
# data now contains:
[[u'Google', u'07/11/2001'],
 [u'Apple', u'27/08/2001'],
 [u'Microsoft', u'01/11/1991']]

# If the data may contain extraneous whitespace you can clean it up
# Additional processing could also be done - but once you hit much more
# complex than this later maintainers, yourself included, will thank you
# for using a series of for loops that call clearly named functions to perform
# the work.
data = [[u"".join(d).strip() for d in l] for l in data]

# If you want to store it joined as name | company
# then simply follow that up with:
data = [u"|".join(d) for d in data]

The list comprehension is basically a reverse for loop with aggregation:

[[td.findNext(text=True) for td in tr.findAll("td")] for tr in rows]

translates into^*:

final_list = []
intermediate_list = []

for tr in rows:
    for td in tr.findAll("td")
        intermediate_list.append(td.findNext(text=True))

    final_list.append(intermediate_list)
    intermediate_list = []

data = final_list

^* Roughly - we are leaving out the awesomeness involving generators not building intermediate lists, since I can't add generators right now without cluttering the example.

edited Nov 15 '11 at 22:27

answered Nov 15 '11 at 16:45

Sean Vieira

155,703
32
311
293

Awesome! Exactly what I was looking for. Though it looks a little complicated, I like one liners. Thank you. – ThinkCode Nov 15 '11 at 16:52
1

@ThinkCode - glad I could help! Just added a bit of explanation of the one-liner for you :-) – Sean Vieira Nov 15 '11 at 16:54
1

Thanks for the explanation Sean. Python sure is awesome! – ThinkCode Nov 15 '11 at 17:18
for val in data: print val doesn't work right? How do I split them for database entry?! Edit : Used this instead! – ThinkCode Nov 15 '11 at 17:23
@ThinkCode - If you are using the lists, you can just access them as `val[0]` and `val[1]` - alternately, you can unpack it by doing `for company_name, date_incorporated in data: print company_name, ":", date_incorporated`. – Sean Vieira Nov 15 '11 at 17:27
interestingly, val[0], val[1] didn't work. Nor does for company_name, date_incorporated in data: print company_name, ":", date_incorporated. This works though : data = [u"|".join(d) for d in data] – ThinkCode Nov 15 '11 at 17:29
Hi Sean, I modified the input file to include a 3rd data point which doesn't always occur. It is wrapped in tags and beautifulsoup is fetching it as 2 blank lines. Do I have to do something different to make it work? Thanks! – ThinkCode Nov 15 '11 at 20:39
@ThinkCode - I'd ask a new question about that :-) – Sean Vieira Nov 15 '11 at 21:04
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/5023/discussion-between-thinkcode-and-sean-vieira) – ThinkCode Nov 15 '11 at 21:07
@SeanVieira When I use your answer I get an "extra" level of nested list like `[[[u'Google'], [u'07/11/2001']], [[u'Apple'], [u'27/08/2001']], [[u'Microsoft'], [u'01/11/1991']]]` – Hugo Apr 17 '16 at 17:36

score 2 · Answer 2 · answered Nov 15 '11 at 16:53

2

Here is small variation of Sean answer if you need exactly what you wrote in question,

table = soup.find("table", id = "history")

rows = table.findAll('tr')

data = ['|'.join([td.findNext(text=True) for td in tr.findAll("td")]) for tr in rows]
print data

answered Nov 15 '11 at 16:53

Gagandeep Singh

5,755
4
41
60

I added a pipe thinking that I can split while storing the data. Sean's solution has the data pairs which saves the splitting cost I believe. Thank you. – ThinkCode Nov 15 '11 at 16:57

How do I extract table data in pairs using BeautifulSoup?

2 Answers2

Linked