1

I'm currently getting price data from Yahoo Finance through a simple web-scraper, but it's taking 1.3-1.5 seconds per request. Ideally, I would like this to be < 1 second, so is there a way to make this faster? I'm just learning Python, so maybe there's a more optimized version of this, but I can't seem to find anything.

import requests
import bs4
from bs4 import BeautifulSoup

def getYahooPrice():
    r = requests.get("https://finance.yahoo.com/quote/AAPL?p=AAPL")
    soup = BeautifulSoup(r.text,"lxml")
    stocks = soup.find_all('div', class_ = 'My(6px) Pos(r) smartphone_Mt(6px)')[0].find('span').text
    print(stocks)

getYahooPrice()
Lehman2020
  • 637
  • 7
  • 22
  • You are doing two steps here. First using `requests.get()` which potentially may be taking most of the 1.3 to .1.5 seconds. Then you are parsing the xml with BeautifulSoup...which I kind of doubt is the slow part. Can you check how fast the request is by itself? – Andrew Allaire Jun 11 '20 at 19:46
  • The request is 1.15s by itself @AndrewAllaire. Is this normal? – Lehman2020 Jun 11 '20 at 19:47
  • Well if you want to speed up the parsing part, you should probably use a faster xml parser. This question may be of use: https://stackoverflow.com/questions/324214/what-is-the-fastest-way-to-parse-large-xml-docs-in-python – Andrew Allaire Jun 11 '20 at 19:50
  • 1
    @Lehman2020 use the websocket directly if you want to optimize: checkout [this post](https://stackoverflow.com/questions/59321908/decode-websocket-received-data) – Bertrand Martel Jun 11 '20 at 20:00

0 Answers0