Python BeautifulSoup scrape tables

Question

I am trying to create a table scrape with BeautifulSoup. I wrote this Python code:

import urllib2
from bs4 import BeautifulSoup

url = "http://dofollow.netsons.org/table1.htm"  # change to whatever your url is

page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)

for i in soup.find_all('form'):
    print i.attrs['class']

I need to scrape Nome, Cognome, Email.

urllib2 is python 2, and has been noted that urllib has security vulnerabilities - best to use the requests package for a higer level api — Aaron Gorman, Feb 04 '22 at 16:33

alecxe · Accepted Answer · 2013-09-23T20:42:34.343

50

Loop over table rows (tr tag) and get the text of cells (td tag) inside:

for tr in soup.find_all('tr')[2:]:
    tds = tr.find_all('td')
    print "Nome: %s, Cognome: %s, Email: %s" % \
          (tds[0].text, tds[1].text, tds[2].text)

prints:

Nome:  Massimo, Cognome:  Allegri, Email:  Allegri.Massimo@alitalia.it
Nome:  Alessandra, Cognome:  Anastasia, Email:  Anastasia.Alessandra@alitalia.it
...

FYI, [2:] slice here is to skip two header rows.

UPD, here's how you can save results into txt file:

with open('output.txt', 'w') as f:
    for tr in soup.find_all('tr')[2:]:
        tds = tr.find_all('td')
        f.write("Nome: %s, Cognome: %s, Email: %s\n" % \
              (tds[0].text, tds[1].text, tds[2].text))

edited Sep 23 '13 at 20:42

answered Sep 23 '13 at 18:40

alecxe

462,703
120
1,088
1,195

can you clarify why you need the [2:] in your first line? – AZhao Apr 11 '16 at 14:24
@AZhao sure, it is there in the answer - to skip the 2 header rows. – alecxe Apr 11 '16 at 14:25

Rakesh Kumar · Answer 2 · 2019-10-04T12:14:54.773

# Libray
from bs4 import BeautifulSoup

# Empty List
tabs = []

# File handling
with open('/home/rakesh/showHW/content.html', 'r') as fp:
    html_content = fp.read()

    table_doc = BeautifulSoup(html_content, 'html.parser')
    # parsing html content
    for tr in table_doc.table.find_all('tr'):
        tabs.append({
            'Nome': tr.find_all('td')[0].string,
            'Cogname': tr.find_all('td')[1].string,
            'Email': tr.find_all('td')[2].string
            })

    print(tabs)

emehex · Answer 3 · 2020-10-10T13:52:59.880

The original link posted by OP is dead... but here's how you might scrape table data with gazpacho:

Step 1 - import Soup and download the html:

from gazpacho import Soup

url = "https://en.wikipedia.org/wiki/List_of_multiple_Olympic_gold_medalists"
soup = Soup.get(url)

Step 2 - Find the table and table rows:

table = soup.find("table", {"class": "wikitable sortable"}, mode="first")
trs = table.find("tr")[1:]

Step 3 - Parse each row with a function to extract desired data:

def parse_tr(tr):
    return {
        "name": tr.find("td")[0].text,
        "country": tr.find("td")[1].text,
        "medals": int(tr.find("td")[-1].text)
    }

data = [parse_tr(tr) for tr in trs]
sorted(data, key=lambda x: x["medals"], reverse=True)

Python BeautifulSoup scrape tables

3 Answers3

Linked