Scraping data with BS4 in Python, nested table

Question

I'm trying to scrape some data from baseball-reference.com. I've written some code to get the data from other parts of the site, where the tables are coded a little more simply, but the particular set of pages is apparently more complicated. Here's the code I have so far.

from urllib.request import urlopen from bs4 import BeautifulSoup

# Declare URL
test_url = 'https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml'

# Query the website and return the HTML
page = urlopen(test_url)

# Parse the HTML and store
soup = BeautifulSoup(page, 'lxml')

table = soup.find("div", {"class": "table_outer_container"})

This doesn't find the tables that I want though (on this particular page, the two tables with At-Bats, RBIs, HRs, runs, etc.). I've tried a few other things, e.g.

table = soup.find_all("table" , {"class": "sortable stats_table"})

but it doesn't work either. I've also tried to read the site using pandas, with no luck, so if there's an easier way with pandas, I'm open to that too.

It looks like an entire part of the page is coded in HTML as a comment. I've seen this before (presumably as a way to get around scrubbing?) but am not sure of the workaround. — Brad Solomon, Jan 21 '18 at 08:35

score 1 · Accepted Answer · answered Jan 21 '18 at 16:00

I know this code is complicated or bad. But it gets the job done. You can optimize it over time.

from bs4 import BeautifulSoup,Comment
import requests

r = requests.get('https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml')
soup = BeautifulSoup(r.text, 'lxml')

comments = soup.find_all(string=lambda text:isinstance(text, Comment))
for comment in comments:
    comment.extract()

    #After getting rid of comments we need Soup again
    another_soup = BeautifulSoup(str(comment),'lxml')

    tables = another_soup.find_all('table' ,{"class": "sortable stats_table"})

    for table in tables:
        #Since we can't get id from table we are going to use table header as stat type.
        stat_type = ''
        for data in table.find('thead').find_all('tr'):
            stat_type = data.th.text.strip()

        #You only need batting.
        if stat_type != 'Batting': break

        for data in table.find('tbody').find_all('tr'):
            player = data.th.text.strip()
            stats = data.find_all('td')

            stat_ab = stats[0].text
            stat_r = stats[1].text
            stat_h = stats[2].text
            stat_rbi = stats[3].text
            # Table goes on
            print(player,stat_ab,stat_r,stat_h,stat_rbi)

        print('-------------------------------------')

The output is:

Kyle Schwarber LF 3 0 2 0
Kris Bryant 3B 4 0 0 0
Anthony Rizzo 1B 4 0 1 0
Ben Zobrist RF 3 1 0 0
Addison Russell SS 4 0 1 0
Jason Heyward CF 4 1 1 0
Willson Contreras C 4 1 2 3
Jon Lester P 2 0 0 0
Carl Edwards P 0 0 0 0
Koji Uehara P 0 0 0 0
Tommy La Stella PH 1 0 0 0
Pedro Strop P 0 0 0 0
Jon Jay PH 1 0 0 0
Mike Montgomery P 0 0 0 0
Javier Baez 2B 4 0 1 0
-------------------------------------
Dexter Fowler CF 4 1 1 0
Aledmys Diaz SS 5 0 2 0
Matt Carpenter 1B 4 0 1 1
Jhonny Peralta 3B 4 0 1 0
Seung-hwan Oh P 0 0 0 0
Jose Martinez PH 1 1 1 0
Yadier Molina C 3 0 2 0
Stephen Piscotty RF 3 1 1 0
Jedd Gyorko 2B 2 0 0 0
Kolten Wong PH-2B 2 0 0 0
Randal Grichuk LF 4 1 2 3
Carlos Martinez P 3 0 0 0
Greg Garcia 3B 0 0 0 0
-------------------------------------

Thanks, this should work as long as I can get this output on my computer. I tried to run it but I don't have Comment installed and when I try pip install comment I get an error. I've tried to searching everywhere to fix it with no luck. Making a new question about that. — jbf, Jan 23 '18 at 23:27
Edit: Nevermind, I didn't realize that Comment was a package inside BS4 itself. That was several hours wasted. This works, thank you. — jbf, Jan 23 '18 at 23:33

score 0 · Answer 2 · answered Jan 21 '18 at 17:12

This is another way you can get the data from the two tables:

import requests
from bs4 import BeautifulSoup, Comment

res = requests.get("https://www.baseball-reference.com/boxes/SLN/SLN201704020.shtml",headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text, 'lxml')
for comment in soup.find_all(string=lambda text:isinstance(text,Comment)):
    data = BeautifulSoup(comment,"lxml")
    for items in data.select("#ChicagoCubsbatting tr,#StLouisCardinalsbatting tr"):
        tds = ' '.join([' '.join(item.text.split()) for item in items.select("th,td")])
        print(tds)

Partial output:

Batting AB R H RBI BB SO PA BA OBP SLG OPS Pit Str WPA aLI WPA+ WPA- RE24 PO A Details
Kyle Schwarber LF 3 0 2 0 0 1 4 .667 .750 1.000 1.750 20 10 0.170 1.75 0.196 -0.026 1.1 2 0 2B,HBP
Kris Bryant 3B 4 0 0 0 0 3 4 .000 .000 .000 .000 19 13 -0.260 2.31 0.000 -0.260 -1.6 0 0 
Anthony Rizzo 1B 4 0 1 0 0 1 4 .250 .250 .250 .500 14 8 -0.214 2.74 0.035 -0.249 -0.9 5 0

Scraping data with BS4 in Python, nested table

2 Answers2