How to let a webpage load before scraping with BeautifulSoup?

Asked Sep 11 '17 at 03:30

Active Sep 11 '17 at 03:30

Viewed 1,752 times

I'm trying to make a small Snow Day Calculator python program, and I figured that I could use BeautifulSoup to do the job. I am able to successfully scrape the page, HOWEVER I noticed that the result is always 0%. I then noticed that this is because it takes the website about a second or two after it loads to display the percentage, and this is because of the website's reactive javascript.

How would I be able to load the webpage, wait two-three seconds, and then scrape the webpage for the snow day percentage?

Code:

import urllib
import bs4

def getPercent(zip):
    url = "https://snowdaypredictor.com/result.html?q="+zip
    print url
    page = urllib.urlopen(url)
    soup = bs4.BeautifulSoup(page,'html.parser')
    name_box = soup.find('span', attrs = {'id': 'result-percent'})
    print name_box.text.strip(),"%"

getPercent(raw_input("What is your zip code?"))

asked Sep 11 '17 at 03:30

Davis Keene

you need https://github.com/scrapinghub/splash – Sraw Sep 11 '17 at 04:21
1

Selenium python bindings might also be helpful. http://selenium-python.readthedocs.io/getting-started.html – Ahmed Dhanani Sep 11 '17 at 04:58
Possible duplicate of [Reading dynamically generated web pages using python](https://stackoverflow.com/questions/13960567/reading-dynamically-generated-web-pages-using-python) – Andersson Sep 11 '17 at 08:43
I was able to get it working with selenium, thank you so much! – Davis Keene Sep 11 '17 at 23:42

How to let a webpage load before scraping with BeautifulSoup?

0 Answers0

Linked