0

I am trying to web scrape this site in order to get basic stock information: https://www.macrotrends.net/stocks/charts/AAPL/apple/financial-ratios

My code is as follows:

from requests import get
from bs4 import BeautifulSoup as bs

url =  'https://www.macrotrends.net/stocks/charts/AAPL/apple/financial-ratios'
response = get(url)
html_soup = bs(response.text, 'html.parser')

stock_container = html_soup.find_all("div", attrs= {'id': 'row0jqxgrid'})


print(len(stock_container))

Right now I am taking it slow and just trying to return the number of "div" under the id name "row0jqxgrid". I am pretty sure everything up to line 8 is fine but I don't know how to properly reference the id using attrs, or if that's even possible.

Can anybody provide any information?

Ross

Ross Leavitt
  • 119
  • 4
  • 13
  • This is because the html content of the table is being generated using java scripts by the browser, hence the html returned by request module does not contain the html code but the java script code for generating it. A work around for this is to use a browser based scrapping tools like selenium. Check out this link for more [info](https://stackoverflow.com/questions/29996001/using-python-requests-get-to-parse-html-code-that-does-not-load-at-once) – Nischal Sanil Feb 03 '20 at 05:23

1 Answers1

0

You can use selenium for this job:

from selenium import webdriver
import os


# define path to chrome driver
chrome_driver = os.path.abspath(os.path.dirname(__file__)) + '/chromedriver'
browser = webdriver.Chrome(chrome_driver)
browser.get("https://www.macrotrends.net/stocks/charts/AAPL/apple/financial-ratios")

# get row element
row = browser.find_element_by_xpath('//*[@id="row0jqxgrid"]')

# find all divs currently displayed
divs_list = row.find_elements_by_tag_name('div')

# get text from cells
for item in divs_list:
    print(item.text)

Output:

Output text is doubled because table data ale loaded dynamically as you move bottom scroll to right.

Current Ratio
Current Ratio
1.5401
1.5401
1.1329
1.1329
1.2761
1.2761
1.3527
1.3527
1.1088
1.1088
1.0801
1.0801
Zaraki Kenpachi
  • 5,510
  • 2
  • 15
  • 38