I'm trying to scrape a table web-site with different table option values

Question

I'm trying to scrap a table web-sites (an example is in this url: https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link ). But there are different option values in this table.

In particular I Would like to set the "Quarter View" option in: Q4 2021 13F Filings

The currently code implementing are:

test = {}
dict_scr = {}
for ii in range (0,12):
    options = webdriver.FirefoxOptions()
    options.binary_location = r'C://Local/Mozilla Firefox/firefox.exe'
    driver = selenium.webdriver.Firefox(executable_path='C:/\geckodriver.exe' , options=options)
    driver.execute("get", {'url': link_scr['Links'][ii]})
    test[link_scr.index[ii]] = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table#current_holdings_table"))).get_attribute("outerHTML")
    dict_scr[link_scr.index[ii]]  = pd.read_html(test[link_scr.index[ii]])
    print(test[link_scr.index[ii]])

How I can update my code for including the option value required?

I don't see your code trials to either of your questions toselect **50** or **Q1 2022 13F Filings**. Am I missing something? — undetected Selenium, Aug 05 '22 at 12:17
I'am searching it! How I can add this option values in my code ? — Agustos Imola, Aug 05 '22 at 12:21

undetected Selenium · Accepted Answer · 2022-08-05T12:43:49.890

-1

To select the <option> with text as Q1 2022 13F Filings you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.execute("get", {'url': 'https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.dfwid-close"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "label#qtr-1-label"))))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#quarter_one")))).select_by_value('85')

Using XPATH:

driver.execute("get", {'url': 'https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='dfwid-close']"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//label[@id='qtr-1-label']"))))
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='quarter_one']")))).select_by_value('85')

Note : You have to add the following imports :

from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser snapshot:

Quarter to view

edited Aug 05 '22 at 12:43

answered Aug 05 '22 at 12:35

undetected Selenium

183,867
41
278
352

It is wrong. Python return me: "WebDriverException: Failed to write request to stream" – Agustos Imola Aug 05 '22 at 12:41
Nada.. the reply is: ElementClickInterceptedException: Element is not clickable at point (857,207) because another element obscures it – Agustos Imola Aug 05 '22 at 13:02
`` is the close icon in Email popup which possibly isn't visible from your locale. Comment out the step if not needed or handle manually to reach to your main query. – undetected Selenium Aug 05 '22 at 13:14
I can only see this in html Inspect: – Agustos Imola Aug 05 '22 at 13:18
The quastion is: how can I scraping this web site, with different option values? for example i need to scraping the first 50 rows of this table and i would like to get the oterh quarter – Agustos Imola Aug 05 '22 at 13:20
Comment out the secondline `WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='dfwid-close']"))).click()` I'm seeing this as I'm out of Europe. – undetected Selenium Aug 05 '22 at 13:20
Is It Clear my question? – Agustos Imola Aug 05 '22 at 13:20
OK. Now the code run correctly but: How can i see the screaping dataframe? And First of All how can i Scraping the first 50 rows? (this is a option values) – Agustos Imola Aug 05 '22 at 13:26
Please restrict from asking multiple questions in a single discussion. Else it won't be useful to the future visitors. – undetected Selenium Aug 05 '22 at 13:30
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247059/discussion-between-agustos-imola-and-undetected-selenium). – Agustos Imola Aug 05 '22 at 13:35

I'm trying to scrape a table web-site with different table option values

1 Answers1