1

I recently tried picking up web scraping on Python and wanted to experiment by grabbing the current temperature in whichever city the user inputs (as long as its in Ontario). The issue I am having is that the return value is empty, so when I check the value of my temperature variable by printing it, it doesnt print anything. The code is below:

import bs4
import requests 

city = input("What city are you in? ")
URL = 'https://www.theweathernetwork.com/ca/weather/ontario/' + city.lower()
website = requests.get(URL)
soup = bs4.BeautifulSoup(website.text,"lxml")
temperature = soup.select(".temp")[0]
print(temperature.text)

Any help would be greatly appreciated, thanks!

Thanatos
  • 11
  • 1
  • That happens because the page content hasn't load yet. So it comes as an empty sting. You should check for solutions to wait the page content load, like [here](https://stackoverflow.com/questions/45448994/wait-page-to-load-before-getting-data-with-requests-get-in-python-3). – Arthur Pereira Nov 20 '20 at 02:27
  • @ArthurPereira That's not the issue. The issue is that the page is dynamic. As in the post you've linked: *It doesn't look like a problem of waiting, it looks like the element is being created by JavaScript, requests can't handle dynamically generated elements by JavaScript.* – Red Nov 20 '20 at 02:31

1 Answers1

0

That page is dynamic, so you won't be able to accomplish it using requests.

Just try print("class='temp'" in soup.text), you will get False.

Instead, use selenium:

from selenium import webdriver
browser = webdriver.Firefox()
city = input("What city are you in? ")
browser.get("https://www.theweathernetwork.com/ca/weather/ontario/" + city.lower())
print(browser.find_elements_by_xpath('.//span[@class = "temp"]')[0].text)

Input:

What city are you in? Waterloo

Output:

12

Just make sure you have gecko driver installed, and you've placed the executable into the same directory as the python file.

Red
  • 26,798
  • 7
  • 36
  • 58