0

I am trying to web scrape this website: https://www.reuters.com/companies/tsla.oq/financials/income-statement-quarterly

I am using Python and everything can be scraped except the date part... i.e. I can't scrape '30-Jun-20'. I tried like

from requests import get
from bs4 import BeautifulSoup
url = 'https://www.reuters.com/companies/tsla.oq/financials/income-statement-quarterly'
response = get(url)
html_soup = BeautifulSoup(response.text, 'html.parser')
table = html_soup.find_all('div', class_ = 'tables-container')
table[0].thead.tr.find('time', class_ = 'TextLabel__text-label___3oCVw TextLabel__black___2FN-Z TextLabel__medium___t9PWg').text

But it shows blank... Can you please help me? That would be much appreciated.

Steve.Kim
  • 71
  • 6

1 Answers1

2

You cannot get the data from a website using requests whose data is dynamically added (using javascript). You need to use selenium to achieve that.

refer this code:

from selenium import webdriver
from bs4 import BeautifulSoup
DRIVER_PATH="Your selenium chrome driver path"
url = 'https://www.reuters.com/companies/tsla.oq/financials/income-statement-quarterly'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get(url)
html_soup = BeautifulSoup(driver.page_source, 'html.parser')
table = html_soup.find_all('div', class_ = 'tables-container')
driver.quit()
print(table[0].thead.tr.find('time', class_ = 'TextLabel__text-label___3oCVw TextLabel__black___2FN-Z TextLabel__medium___t9PWg').text)
Yash
  • 1,271
  • 1
  • 7
  • 9
  • Thank you very much for your reply. But I don't know what is "Your selenium chrome driver path". Can you please explain me how to get this path? FYI, I am using Google Colab to run the script. – Steve.Kim Sep 16 '20 at 01:27
  • @stevekim Hi, what i mean is you need to download selenium chrome webdriver from [link](https://chromedriver.chromium.org/downloads) version should be your current chrome version. else it will throw an error. add the path to exe in driver_path. Since you are doing it on google colab, answer on this [link](https://stackoverflow.com/questions/51046454/how-can-we-use-selenium-webdriver-in-colab-research-google-com) should help you. – Yash Sep 16 '20 at 07:17