1

I'm trying to scrape this website http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae but when I want to go to next page I can't because the link doesn't change you will find that pages links are like that 

href="javascript:__doPostBack('sb$grd','Page$1')"

I have a code that I tried but it only goes to page 2 and then gave me an error: tale element reference: element is not attached to the page document

from selenium import webdriver
url = 'http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae'
driver = webdriver.Chrome()
driver.get(url)
btn = [w for w in driver.find_elements_by_xpath('//*[@id="sb_grd"]/tbody/tr[26]/td/table/tbody/tr/td/a')]
for b in btn:
    driver.execute_script("arguments[0].click();", b)
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Hosam Gamal
  • 163
  • 1
  • 2
  • 12
  • There's a form on the page (`name="form1"`), which is submitted when the link is activated. You could post such a form to the site, but if the request is not responded (or the response is the actual page), I suppose you're out of luck. – Teemu Dec 17 '19 at 12:47

2 Answers2

1

To paginate through the page numbers with href attribute as "javascript:__doPostBack('sb$grd','Page$2')" you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae")
while True:
    try:
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//table[@id='sb_grd']//table/tbody/tr//td/span//following::td[1]/a"))).click()
        print("Next page clicked")
    except TimeoutException:
        print("No more pages")
        break
driver.quit()

  • Console Output:

    Next page clicked
Next page clicked
Next page clicked
.
.
.
No more pages

Reference

You can find a relevant detailed discussion in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
0

You can see that a request is made under the wood to this url : http://www.mfa.gov.tr/default.fr.mfa Do F12 and go to the network tab to see that.

All the articles you need will be in the document at this url, you don't need to paginate as the pagination has just a purpose display. No request is being made by this pagination, expect to download the displayed image.

SimonR
  • 523
  • 1
  • 5
  • 17