Selenium not performing click action

Question

I am trying to scrape a table with multiple pages. The next page is obtained by clicking on the 'Next Page button' (See code snippet).

<a class="botons" id="btn2" href="javascript:void(0)">
 Next Page  
 <i class="fas fa-long-arrow-alt-right"></i>
</a>

Selenium finds the "button" and has no trouble "clicking" via the following code:

btn_next = self.browser.find_element_by_partial_link_text("Next Page")
btn_next.click()

However, the page just refreshes and the table doesn't update to its next page.

Any clues to what's going wrong here?

Edit: table can be found at https://www.proxy-list.download/HTTPS

Edit2:

chrome_options = Options()
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")

chrome_options.add_argument("--headless")

there are at least two event handlers firing there.... one is on the tag... that one opens a new window and sets it to the same URL. The other one is placed on the button itself, and populates the table but only if a certain var evaluates to true. (this probably happens when the ad loads in the popunder) Maybe try generating a click on the tag and then a click on the button... or execute a javascript to set the var "btn2ena === true" — pcalkins, Jul 11 '19 at 20:03
you might also try just clicking it twice... you'll probably also need to switch the driver to the new tab after this happens. (adding the wait listed in the answers below will also be needed cause the button handler is added on pageload...) — pcalkins, Jul 11 '19 at 20:10
@pcalkins This actually solved it! Thanks a bunch. If you submit your comments as an answer, I'll accept and close this thread. — xrdty, Jul 12 '19 at 16:31
good to hear... go ahead and post your own answer. Not sure which solved your issue. — pcalkins, Jul 12 '19 at 17:38

score 1 · Answer 1 · answered Jul 11 '19 at 11:18

1

There is one id assigned to that button btn2 and it's unique too.

You should give preference to id over link Text.

That said, Next Page link isn't present in view point.For that first you have to move the focus of driver like this :

wait = WebDriverWait(self.browser,10)
next_page = wait.until(EC.visibility_of_element_located((By.ID, "btn2")))
ActionChains(self.browser).move_to_element(next_page).perform()
next_page.click()

Imports :

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

answered Jul 11 '19 at 11:18

cruisepandey

28,520
6
20
38

Thanks for you input, but this doesn't seem to fix the issue. The problem remains the same. – xrdty Jul 11 '19 at 11:47
After running it, a new tab is opening ? – cruisepandey Jul 11 '19 at 11:51
Sometimes a new tab opens for an advertisement. But selenium then still manages to scrape the first page of the table. So I doubt that it's causing any problems? – xrdty Jul 11 '19 at 11:55
in my system, it is opening in new tab after execution, and first tab got blocked, I'm in private network. Ask your dev (If possible ) to give a workaround in QA Env. – cruisepandey Jul 11 '19 at 11:57

score 0 · Answer 2 · answered Jul 11 '19 at 11:43

The desired element is an JavaScript enabled element so to locate and click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.botons[id^='btn2'] i.fas.fa-long-arrow-alt-right"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='botons' and normalize-space()='Next Page']/i[@class='fas fa-long-arrow-alt-right']"))).click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Thanks for you input, but this doesn't seem to fix the issue. The problem remains the same. — xrdty, Jul 11 '19 at 11:47

Selenium not performing click action

2 Answers2