2

Page which i need to scrape data from: Digikey Search result

Issue
It is allowed to show only 100 row in each table, so i have to move between multiple tables using the NextPageButton. As illustrated in the code below, I actually do though, but the results retrieves to me every time the first table results and doesn't move on to the next table results on my click action ActionChains(driver).click(element).perform().
Keep in mind that NO new pages is opened, click is going to be intercepted by some sort of JavaScript to do some rich UI stuff on the same page to load a new table of data

My Expectations
I am just trying to validate that I could move to the next table, then i will edit the code to loop through all of them. This piece of code should return the data in the second table from results, BUT it actually returns the values from the first table which loaded initially with the URL. This means that the click action didn't occur or it actually occurred but the WebDriver driver content isn't being updated by interacting with dynamic JavaScript elements in the page.

I will appreciate any help, Thanks..

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver import ActionChains
import time
import sys


url = "https://www.digikey.com/en/products/filter/coaxial-connectors-rf-terminators/382?s=N4IgrCBcoA5QjAGhDOl4AYMF9tA"

chrome_driver_path = "..PATH\\chromedriver"
chrome_options = Options() 
chrome_options.add_argument ("--headless") 

webdriver = webdriver.Chrome(
    executable_path= chrome_driver_path
    ,options= chrome_options 
)


with webdriver as driver:
    wait = WebDriverWait(driver, 10)
    driver.get(url)
    wait.until(presence_of_element_located((By.CSS_SELECTOR, "tbody")))
    
    element = driver.find_element_by_css_selector("button[data-testid='btn-next-page']")
    ActionChains(driver).click(element).perform()
    time.sleep(10) #too much time i know, but to make sure it is not a waiting issue. something needs to be updated

    results = driver.find_elements_by_css_selector("tbody")

    for count in results:
        countArr = count.text
        print(countArr)
        print()
    
    driver.close()
MSamyGawad
  • 71
  • 6
  • I knwo you have answered your question but let me propose an easier possible alternative. Does `element.click()` work? – Charalamm Dec 13 '20 at 09:10
  • Nope, i tried it before.Also i tried every piece of code in python creating the click action in different ways, but only this Javascript execution method the only one worked for me. maybe because the website i scrape runs tons of javascript stuff. – MSamyGawad Dec 13 '20 at 21:35

1 Answers1

2

Finally found a SOLUTION !

Source of the solution.
As expected the issue was in the clicking action itself. It is somehow not being done right or it's not being done at all as illustrated in the solution Source question.

the solution is to click the button using Javascript execution.
Change line 30 ActionChains(driver).click(element).perform() to be as following:

driver.execute_script("arguments[0].click();",element)

That's it..

MSamyGawad
  • 71
  • 6