Scrape a text that was written by javascript from website

Question

I am using BeautifulSoup to scrape character info from a website. When trying to get the win rate of a character, BeautifulSoup can't find it.

When I inspect the text, it is listed as under . All I can find in the sites source code and all that BeautifulSoup finds is "ranking-stats-placeholder".

This is the code I am currently using.

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = "https://u.gg/lol/champions/darius/build/?role=top"

#opening up connection, grabbing the page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

#html parsing
page_soup = soup(page_html, "html.parser")

#champion name
champ_name = page_soup.findAll("span", {"class":"champion-name"})[0].text

#champion win rate
champ_wr = page.soup.findAll("div", {"class":"win-rate okay-tier"})

I believe that the win rate text is added by javascript, but I have no idea how I can get the text. The code I currently have returns "None" for champ_wr

https://stackoverflow.com/questions/13960567/reading-dynamically-generated-web-pages-using-python — Monish K Nair, Sep 06 '18 at 13:57

score 0 · Answer 1 · answered Sep 06 '18 at 14:07

0

Although this text technically could be in the javascript itself, my first guess is the JS is pulling it in via an ajax request. Have your program simulate that and you'll probably get all the data you need handed right to you without any scraping involved!

It will take a little detective work though. I suggest turning on your network traffic logger (such as "Web Developer Toolbar" in Firefox) and then visiting the site. Focus your attention attention on any/all XmlHTTPRequests.

Best of luck!

answered Sep 06 '18 at 14:07

TheHeadlessSourceMan

587
6
18

I couldn't find any XmlHTTPRequests, but I managed to find that everything I need is in a single .js file. I have no idea how to work with this though... – Kevin Sep 06 '18 at 14:35
If you found what you need, can you directly parse it from the .js file, perhaps using a regex? – TheHeadlessSourceMan Sep 06 '18 at 14:41
1

Chrome's dev tools (Ctrl+Shift+J in Linux) has a "Network" tab that would list these requests… – Sam Mason Sep 06 '18 at 15:01
@SamMason Yes, but I didn't find any XmlHTTPRequests in there – Kevin Sep 06 '18 at 16:48

score 0 · Answer 2 · answered Sep 06 '18 at 21:55

not sure how tied to BeautifulSoup you are, but I can get selenium doing useful things with:

# load code from selenium package
from selenium.webdriver import Remote
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

# start an instance of Chrome up
chrome = Service('/usr/local/bin/chromedriver')
chrome.start()
driver = Remote(chrome.service_url)

# get the page loading
driver.get("https://u.gg/lol/champions/darius/build/?role=top")

# wait for the win rate to be populated
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "win-rate")))

# get the values you wanted
name = driver.find_element_by_class_name("champion-name").text
winrate = driver.find_element_by_class_name("win-rate").text

# display them
print(f"name: {repr(name)}, winrate: {winrate.split()[0]}")

# clean up a bit
driver.quit()

Scrape a text that was written by javascript from website

2 Answers2