Selenium - How to to click all the More buttons of each individual items to scrape the data from the dropdown

Question

I'm trying to scrape the information on a page, but when I open the exported .CSV file it is blank except for the headings.

I'm trying to scrape the 10 results on this page: https://www.narpm.org/find/property-managers/?submitted=true&toresults=1&resultsperpage=10&a=managers&orderby=&fname=&lname=&company=&chapter=S005&city=&state=&xRadius=

I can scrape the name, company, city and state, but when it comes to clicking on the dropdown 'More' it just doesn't seem to work. (Not getting any errors, csv is just blank.)

I suspect the problem lies is this code block:

driver.find_element_by_xpath('//div[@class="col-md-4 col-lg-1 arrow"]').click()

Here is all of my code:

options = Options()
options.headless = True
driver = webdriver.Chrome(executable_path='/Users/vilje/anaconda3/envs/webscrape/chromedriver', options=options)
driver.set_window_size(1440, 900)

# Creates master dataframe
df = pd.DataFrame(columns=['Name','Company', 'City', 'State', 'Phone', 'About']) 

# URL
driver.get('https://www.narpm.org/find/property-managers/?submitted=true&toresults=1&resultsperpage=10&a=managers&orderby=&fname=&lname=&company=&chapter=S005&city=&state=&xRadius=')


name = driver.find_elements_by_xpath('//span[@class="name"]')
company = driver.find_elements_by_xpath('//div[@class="col-md-6 col-lg-4"]')
city = driver.find_elements_by_xpath('//div[@class="col-md-4 col-lg-2"]')
state = driver.find_elements_by_xpath('//div[@class="col-md-4 col-lg-2"]')

# Expand the 'More' button
driver.find_element_by_xpath('//div[@class="col-md-4 col-lg-1 arrow"]').click()
phone = driver.find_elements_by_xpath('//div[@class="col-sm-6 col-lg-3 with-icon lighter-text"]')
about = driver.find_elements_by_xpath('//div[@class="col-sm-12"]')

name_list = []
for n in range(len(name)):
    name_list.append(name[n].text)

company_list = []
for c in range(len(company)):
    company_list.append(company[c].text)

city_list = []
for c in range(len(city)):
    city_list.append(city[c].text)

state_list = []
for s in range(len(state)):
    state_list.append(state[s].text)

phone_list = []
for p in range(len(phone)):
    phone_list.append(phone[p].text)

about_list = []
for a in range(len(about)):
    about_list.append(about[a].text)

# List of each property managers name, company, city, state, phone and about section paired together
data_tuples = list(zip(name_list[0:], company_list[0:], city_list[0:], state_list[0:], phone_list[0:], about_list[0:]))

# Creates dataframe of each tuple in list
temp_df = pd.DataFrame(data_tuples, columns=['Name','Company', 'City', 'State', 'Phone', 'About']) 

# Appends to master dataframe
df = df.append(temp_df)

driver.close()

Can anyone please help me to click all the 'More' buttons of each individual so I can scrape the data from the dropdown.

Answer 1

To click all the elements with text as More you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  driver.get("https://www.narpm.org/find/property-managers/?submitted=true&toresults=1&resultsperpage=10&a=managers&orderby=&fname=&lname=&company=&chapter=S005&city=&state=&xRadius=")
  for more in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.row div.arrow"))):
    more.click()
  

• Using XPATH:

  driver.get("https://www.narpm.org/find/property-managers/?submitted=true&toresults=1&resultsperpage=10&a=managers&orderby=&fname=&lname=&company=&chapter=S005&city=&state=&xRadius=")
  for more in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='row']//div[contains(@class, 'arrow') and contains(., 'More')]"))):
    more.click()
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Browser Snapshot: