Webscraping Live data

Question

I am currently trying to scrape live stock market data from the yahoo finance page.

I am using bs4. My current issue is that whenever I run my script, it does not update properly to reflect the current price of the stock.

If anybody has any advice on how to change that it would be appreciated.

import requests
from bs4 import BeautifulSoup

while True:
    page = requests.get("https://nz.finance.yahoo.com/quote/NZDUSD=X?p=NZDUSD=X")
    soup = BeautifulSoup(page.text, "html.parser")
    price = soup.find("div", {"class": "My(6px) Pos(r) smartphone_Mt(6px)"}).find("span").text
    print(price)

Sharyar Vohra · Accepted Answer · 2020-04-21T09:33:14.250

NOT POSSIBLE WITH BS4 ALONE

This website particularly uses JavaScript to update the page and urlib etc. just parses the html content of the page not Java Script or AJAX content. PhantomJs or Selenium Web Browser provide a more mechanized browser that often can run the JavaScript codes enabling dynamic websites. Try Using this :)

Using Selenium It can be done as:

    from selenium import webdriver   #its the library
        import time
        from selenium.webdriver.common.keys import Keys
        from bs4 import BeautifulSoup as soup

            #it Says that we are going to Use chrome browser
        chrome_options = webdriver.ChromeOptions()
            #hiding the Chrome Browser
        chrome_options.add_argument("--headless")

    #Initiating Chrome with all properties we need (in this case we use no specific properties
        driver = webdriver.Chrome(chrome_options=chrome_options,executable_path='C:/Users/shary/Downloads/chromedriver.exe')
    #URL We need to open
        url = 'https://nz.finance.yahoo.com/quote/NZDUSD=X?p=NZDUSD=X'

    #Starting Our Browser
        driver = webdriver.Chrome()
    #Accessing the url .. this will open the page just as you open in Chrome etc.
        driver.get(url)

        while 1:
    #it will get you the html content repeatedly .. So you can get the changing price
            html = driver.page_source
            page_soup = soup(html,features="lxml")
            price = page_soup.find("div", {"class": "D(ib) Mend(20px)"}).text
            print(price)
            time.sleep(5)

Note the Best Comments But Hope this you will understand it :) Else Watch a youtube tutorial to get proper idea what a Selenium Bot does

Hope This will Help. Its working perfect for me :) Accept This Answer if it helps you

Hi there, just because I'm new to Web scraping can you please either explain your code or add some comments to make it easier to understand what's going on :) — FPOLinux, Apr 20 '20 at 09:55
Comments are added. Mark this as your answer Coz your required answer is given :) — Sharyar Vohra, Apr 20 '20 at 11:12
Hi I made some edits for linux based systems. For chrome driver to work I had to go to https://stackoverflow.com/questions/48213384/how-to-add-chromedriver-to-path-in-linux and look at @FrancescoBorzi solution — FPOLinux, Apr 20 '20 at 11:56
Is it possible to run this program without opening a chrome window? Expecially if i wanted it to be automated — FPOLinux, Apr 20 '20 at 11:58

Webscraping Live data

1 Answers1