2

I'm trying to get wind speeds from the Google 7 day forecast. When I inspect the webpage code I can see the wind speeds but when I use find_all() on the class it returns only temperature data and todays wind speed from the 7 day forecast.

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.google.co.nz/search?ei=CQmzW9_zHsaiwAPuvruwCQ&q=tauranga+weather+forecast&oq=tauranga++forecast&gs_l=psy-ab.3.0.0i7i30k1l10.9062.9062.0.11810.1.1.0.0.0.0.205.205.2-1.1.0....0...1c.1.64.psy-ab..0.1.205....0.R-r6_9AWgnA")
soup = BeautifulSoup(page.content, "html.parser")

wind = soup.find_all("span", class_="wob_t")

for i, e in enumerate(wind):
    print(i, e.get_text())

What am I doing wrong here?

Jongware
  • 22,200
  • 8
  • 54
  • 100
davedavedave123
  • 21
  • 2
  • 1
    If the layout changes your app will break. Consider using a weather API instead. – Royal Wares Oct 02 '18 at 07:09
  • 1
    i'll agree with alexander.. but just for fun do a find_all for div with class "e". you will reach the div where the wind span is – iamklaus Oct 02 '18 at 07:12

1 Answers1

0

I see that the desired div content what you are looking at is generated by some javascript code. i.e: when you click on the "Wind" button displayed on page, javascript modify the HTML and generate the wind div for 7 days.

With the soup you generated with the given URL, I see there is only a single entry of wind speed div. This is something which is shown on the page. 

In [7]: soup.findAll("span", text = re.compile("km/h"))
Out[7]: [<span class="wob_t" style="display:inline">16 km/h</span>]

So, scraping these type of page which update HTML based on some javascript modification won't be a good idea to scrap using request module. I would suggest accessing these types of the page you should use python-selenium.

Vidya Sagar
  • 1,496
  • 14
  • 23
  • Ok thanks, I had thought selenium might be outside of my skill level but I'll give it a go. I can see all the wind data for the 7 day forecast in the source code: 13 km/h is this then modified by javascript? Thanks for your help – davedavedave123 Oct 02 '18 at 07:35