Web Scraping WeatherUnderground

Question

So I'm doing the project that reads the data every hour from WeatherUnderground and it saves the data into a txt file

from selenium import webdriver
from selenium.webdriver.common.by import By
from datetime import datetime

def scrape_weather_data(url, time_selector, temperature_selector, 
humidity_selector, precip_selector, wind_speed_selector, 
pressure_selector, ammount_selector):
driver = webdriver.Chrome()  

driver.get(url)  # Enter The website

# Generate text file name based on day and hour
current_time = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
filename = f'dane_pogodowe_{current_time}.txt'

# Saving data into a .txt file
with open(filename, 'a') as file:
    time_elements = driver.find_elements(By.CSS_SELECTOR,time_selector)
    temperature_elements = driver.find_elements(By.CSS_SELECTOR,temperature_selector)
    humidity_elements = driver.find_elements(By.CSS_SELECTOR,humidity_selector)
    precip_elements = driver.find_elements(By.CSS_SELECTOR,precip_selector)
    wind_elements = driver.find_elements(By.CSS_SELECTOR, wind_speed_selector)
    pressure_elements = driver.find_elements(By.CSS_SELECTOR, pressure_selector)
    ammount_elements = driver.find_elements(By.CSS_SELECTOR, ammount_selector)

    for i in range(len(time_elements)):
        time = time_elements[i].text.strip()
        temperature = temperature_elements[i].text.strip()
        humidity = humidity_elements[i].text.strip()
        precip = precip_elements[i].text.strip()
        wind = wind_elements[i].text.strip()
        pressure = pressure_elements[i].text.strip()
        ammount = ammount_elements[i].text.strip()

        file.write(f"Czas: {time}\n")
        file.write(f"Temperatura: {temperature}\n")
        file.write(f"Wilgotność: {humidity}\n")
        file.write(f"Szansa opadów: {precip}\n")
        file.write(f"Wiatr: {wind}\n")
        file.write(f"Ciśnienie: {pressure}\n")
        file.write(f"Ilość opadów: {ammount}\n")
        file.write("\n")
    file.write("------------------------------\n")

driver.quit()  # Close the Web browser

print("Dane pogodowe zostały zapisane.")

# URL
url = 'https://www.wunderground.com/hourly/pl/kucoby'

# Selectors CSS
time_selector = '#hourly-forecast-table > tbody > tr:nth-child(1) 
> td.mat-cell.cdk-cell.cdk-column-timeHour.mat-column- 
timeHour.ng-star-inserted > span'
temperature_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column-temperature.mat- 
column-temperature.ng-star-inserted'
humidity_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column-humidity.mat-column- 
humidity.ng-star-inserted'
precip_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column-precipitation.mat- 
column-precipitation.ng-star-inserted'
wind_speed_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column-wind.mat-column- 
wind.ng-star-inserted'
pressure_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column-pressure.mat-column- 
pressure.ng-star-inserted'
ammount_selector = '#hourly-forecast-table > tbody > tr:nth- 
child(1) > td.mat-cell.cdk-cell.cdk-column- 
liquidPrecipitation.mat-column-liquidPrecipitation.ng-star- 
inserted'


# Calling a function and saving the Data
scrape_weather_data(url, time_selector, temperature_selector, 
humidity_selector, precip_selector, wind_speed_selector, 
pressure_selector, ammount_selector)
`

And the data reading is based on CSS_Selectors as you can tell, but the problem is that when it enters the website, the default setting is that it opens in the imperial system, not the metrical that i use. And i can't really convert it mathematically into metric system, due to the fact that the variables are saved as strings. There is a option on this website to convert data from imperial into metrical system, but to be fair i have no idea, how to implement it.

EDIT I found out the key that is responsible for setting the metric system:

you can convert a string to an int or float with `int()` or `float()` methods. — D.L, Jun 04 '23 at 22:20

score 0 · Answer 1 · answered Jun 04 '23 at 22:24

0

You can convert (typecast) strings to int or float with the int() and float() methods like this:

# convert a string to an integer
string = "123"
int_value = int(string)
print(int_value)


# convert a string to a float
string = "123.123"
float_value = float(string)
print(float_value)

You can then do whatever conversions you like...

this way, you would not need to modify the selenium part of the code (given that it is working).

answered Jun 04 '23 at 22:24

D.L

4,339
5
22
45

Yes that was my first idea, the thing is that it wouldn't work with a wind column that is given as shown "4 mph E", I'd have to seperate it and i was wondering if i could work this out with selenium i tried driver.add_cookie({"name": "wu.Units", "value": "metric"}) but I didnt do much – user22018490 Jun 04 '23 at 23:03
why dont you just toggle to metric system from the UI? – Murat Ishenbaev Jun 05 '23 at 00:59
@user22018490, you can use the `split()` method to separate strings into numbers and units. `split(" ")`. – D.L Jun 05 '23 at 07:09
@MuratIshen because this project is supposed to be automatic - thats at least my proffesor says – user22018490 Jun 05 '23 at 10:16

score 0 · Answer 2 · answered Jun 05 '23 at 01:01

0

you can toggle to metric system from the UI directly in the website top-right corner. hope it helps.

answered Jun 05 '23 at 01:01

Murat Ishenbaev

168
10

Yes, I've added that in the Edit i posted. But how do i to that in code without interacting with UI? Using sellenium only? – user22018490 Jun 05 '23 at 09:32
yes you can toggle it to automatically by selenium. – Murat Ishenbaev Jun 05 '23 at 14:51
how is that??? @Murat Ishen – user22018490 Jun 05 '23 at 18:35
can you provide the code where you try change to metric system? – Murat Ishenbaev Jun 05 '23 at 20:36
i mean i tried to do that with selenium, i sent a new question 5 minutes ago, check this out https://stackoverflow.com/questions/76409811/weatherunderground-web-scraping-choosing-different-measurements-systems – user22018490 Jun 05 '23 at 20:41

Web Scraping WeatherUnderground

2 Answers2