0

I made this little script as I'm learning to scrape with bs4 and selenium and I want to collect periodic date of weather into a csv file but I am quite new and I don't know where to begin.

Can someone point me in the right direction on to how to make this run every date and append data? For now, I run and it just makes the header and the second row.

from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.common.by import By from csv import writer

url = 'https://weather.com/es-US/tiempo/10dias/l/468124beb45cd6817ec6a1da2675f104fede76a75ba7ce08c9ae21606ac208b5'
driver = webdriver.Chrome('/Users/thras/Desktop/Chromedriver/chromedriver')
driver.get(url)
source = driver.page_source

soup = BeautifulSoup(source, 'html.parser')



degrees = soup.find('span', class_="DailyContent--temp--3d4dn").text
humidity = soup.find('span', class_="Wind--windWrapper--3aqXJ DailyContent--value--37sk2").text
wind = soup.find('span', class_="DailyContent--temp--3d4dn").text
date = soup.find('span', class_="DailyContent--daypartDate--2A3Wi").text

info = [degrees, humidity, wind, date]

with open('weather.csv', 'w', encoding='utf8', newline='') as f:
    mywriter = writer(f)
    header = ('Degrees', 'Humidity', 'Wind', 'Date')
    mywriter.writerow(header)
    mywriter.writerow([degrees, humidity, wind, date])
    
driver.quit()
Alain Procs
  • 23
  • 3

2 Answers2

1

To schedule the python script to run it every day you could either use a builtin task scheduler of your os or you could schedule it inside the script itself. Here you could use the python library schedule. You can see an example here: stackoverflow schedule

To add the row to your existing csv, you just have to open it in append mode.

import csv   
fields=['A','B']
with open('weather.csv', 'a') as f:
    writer = csv.writer(f)
    writer.writerow(fields)
gmad
  • 138
  • 9
  • I actually realized I can't append it for some reason, could you show me how would you do it please? – Alain Procs Sep 16 '22 at 01:30
  • I added a small example of writing csv in append mode. The difference ist mostly the 'a' instead of the 'w', as the second parameter. – gmad Sep 16 '22 at 07:21
1

I think all you need to do is to schedual your programm to be executed every day. If you are on MacOs or Linux you can use crontab.

If you are on Windows you can use Windows Task Scheduler search on the interent how they work.

HamzaDevXX
  • 154
  • 1
  • 10