Using Selenium to click specific links in a table

Question

I have to collect data from this link: https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible

However I am having difficulty accessing the data under substitutes(note: not all them have substitutes). Those with substitutes look like this:

An example is on page 2 of the link

Kindly help me finish the code I have to collect the part numbers of the substitutes.

Here's my code:

from selenium import webdriver
from time import sleep
import csv

# initializing webdriver 
driver = webdriver.Chrome(executable_path="~~chromedriver.exe")
url = "https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible"
driver.get(url)
sleep(5)

#getting breadcrumbs
bread1 = driver.find_element_by_xpath("//span[@class='prod-catagory-name']")
bread2 = driver.find_element_by_xpath("//span[@class='prod-catagory-name']/a")

#grabbing table data and navigating 
pages = int(driver.find_element_by_xpath("//div[@class='page-container']/span[@class='icon-s-right active']/preceding-sibling::span[1]").text)
num = pages -1 
for _ in range(num):
     rows = driver.find_elements_by_xpath("//table/tbody/tr/td[2]/div")
     for row in rows:
         parts = row.text
         with open(filename, 'a', encoding='utf-8') as file:
             file.write(url + "," + bread1.text + "," + bread2.text + "," + parts + "\n")

     pagination = driver.find_element_by_xpath("//div[@class='pagecontainer']/span[@class='icon-s-right active']").click()
     sleep(5)
driver.close()

Please let me know if I need to change or modify the code to get the substitutes.

score 1 · Accepted Answer · answered Nov 26 '21 at 22:15

1

To access the data under substitutes within the webpage you need to expand the Substitutes inducing WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Code Block:

driver.get("https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.icon-select-down"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[contains(., '20')]"))).click()
for element in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span[class='icon-s-down']"))):
    element.click()
    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='icon-s-up']//following::tr[3]/td[contains(@class,'enabled-border')]//div[text()]"))).text)

Console Output:

Lenovo 30m LC-LC OM3 MMF Cable
Lenovo 30m LC-LC OM3 MMF Cable
5m LC-LC OM3 MMF Cable
5m LC-LC OM3 MMF Cable

answered Nov 26 '21 at 22:15

undetected Selenium

183,867
41
278
352

1

Thanks. So any idea how to incorporate this into my original code? Because I need all the details including the substitutes.. – Reggie18 Nov 27 '21 at 14:12
1

Thanks, I found a way to incorporate it into my code. – Reggie18 Nov 27 '21 at 15:16
1

I placed it in a try-except block – Reggie18 Nov 27 '21 at 15:32
thanks. Small issue though. I haven't been able to export them to a csv file. Given all the code I have posted, how do I export the data to a csv file if there are more than one substitute in each line. Like in this link – Reggie18 Nov 28 '21 at 22:55
@Reggie18 Feel free to raise a new question as per your new requirement. – undetected Selenium Nov 29 '21 at 09:39
Cool...... Please take a look at the question here[https://stackoverflow.com/questions/70162102/export-multiple-scraped-items-to-a-csv-file-in-selenium] – Reggie18 Nov 29 '21 at 22:32

Using Selenium to click specific links in a table

1 Answers1