How to scrape all the search results using Selenium webdriver and Python

Question

I'm trying to scrape all CRD# from the search result from this site https://brokercheck.finra.org/search/genericsearch/list

(You'll need to redo the search when you click on the link, just type some random stuff for the Individual search)

I'm using driver.find_elements_by_xpath to target all CRD numbers on each result page. However, I've been playing around with the paths for a while but the webdriver still can't pick up the CRDs from the site.

I currently have (in Python)

crds = driver.find_elements_by_xpath("//md-list-item/div/div/div/div/div/bc-bio-geo-section/div/div/div/div/div/span")

But the result is always empty.

They have similar span classes of spacerright ng-binding. driver.find_elements_by_xpath("//span[@class = 'spacerright ng-binding']")); Then loop and print .getText() maybe? — Arundeep Chohan, May 19 '20 at 03:55

score 1 · Answer 1 · answered May 19 '20 at 05:10

1

Try to use .find_elements_by_css_selector like this:

crds = driver.find_elements_by_css_selector("span[ng-bind-html='vm.item.id']")

answered May 19 '20 at 05:10

frianH

7,295
6
20
45

score 1 · Accepted Answer · answered May 19 '20 at 18:22

To print all the CRD# from the search results within the website https://brokercheck.finra.org/search/genericsearch/grid using Selenium you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and get_attribute():

print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ng-binding[ng-bind-html='vm.item.id']")))])

Using XPATH and text:

print([my_elem.text for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[starts-with(., 'CRD')]//following-sibling::span[1]")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to scrape all the search results using Selenium webdriver and Python

2 Answers2

Linked