1

I'm scraping a page using selenium and python. The data is paginated and the table data looks like below.

<td>
    <a href="javascript:__doPostBack('ac$w$PC$PC$grid','Page$2')">2</a>
</td>

The challenge now is to get selenium to click on this link and advance to the next page.

There's a SO question that tries to address the problem but its not in python.

There are also a lot of SO questions that try to address this using execute_script but none of them addresses the added complication of a javascript function with seemingly two arguments.

How do I get selenium to click on this link and advance to the next page? Any help or pointer is highly appreciated.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
chidimo
  • 2,684
  • 3
  • 32
  • 47
  • Selenium "clicks" the link: **so "click" the link, as if it were any other link**. The href being 'javascript:blah' doesn't matter as Selenium runs in the browser context, "just like a normal user". Selenium supports XPath selectors, and having a name/id on the element would probably make the task 'easier'. – user2864740 Sep 18 '18 at 16:59
  • Just click the link and it will advance to the next page. Please post your code and what isn't working. Right now it looks like you are just asking for a locator to click a link that contains the text '2' and that question has already been asked many times, `By.link_text('2')` will work as will an XPath. – JeffC Sep 18 '18 at 18:22

2 Answers2

1

You need to induce WebDriverWait for the staleness of the element and can use the following solution:

WebDriverWait(driver, 10).until(EC.staleness_of(driver.find_element_by_xpath("//td/a[text()='2']")))
driver.find_element_by_xpath("//td/a[text()='2']").click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You will find a relevant discussion in How do I wait for a JavaScript __doPostBack call through Selenium and WebDriver

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    When you use `contains()` on something like '2', you are opening up the locator to ANY link inside a table that contains a 2... 12, 20, 21, 22, ...102, 'some string of text that contains a 2 anywhere', and so on. It would be much better if you used `//td/a[.='2']` – JeffC Sep 18 '18 at 18:11
  • You missed the update for the first line. Since you are using the same locator twice, it would probably be better to declare the locator as a variable and then use the variable. This won't work btw since you are clicking *after* waiting for stale. The wait for stale will timeout since nothing has happened to make it go stale. You need to get the element and assign it to a variable, `link`... then do `link.click()`, then wait for `link` to go stale. – JeffC Sep 18 '18 at 18:19
  • I don't think this is even addressing the question... I think the question is more how do I click on this link (needs a locator to find the element)... nothing to do with clicking and waiting for stale. – JeffC Sep 18 '18 at 18:20
  • this answer wouldn't work.. the first line waits for the element to become stale, then it tries to find/click the stale element (which is impossible). – Corey Goldberg Sep 18 '18 at 21:35
1

This should work:

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

[...]

WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH,"//td/a[text()='2']")))
driver.find_element_by_xpath("//td/a[text()='2']").click()
GabrielaGU
  • 11
  • 1