Scraping loop on list with next page using Selenium

Question

is somewhat similar althogh i dont want to make it more complicated by using with bs4

I want to scrape all 10 titles (links to pdfs) on every page. Here is the code that throws a StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (at i.click())

driver.find_element_by_css_selector('button[class="btn btn-primary"][type="submit"]').click()    
    
for j in range(10):
    pdfs = driver.find_elements_by_css_selector(".title a")
    if j < 10:
        nextpg = driver.find_element_by_css_selector(".tx-pagebrowse-page button")
        pdfs = driver.find_elements_by_css_selector(".title a")
    for i in pdfs:
        i.click()
        time.sleep(5)
    if j < 10:
        nextpg.click()
    else:
        break

below it shows that the first page is a type=list list with the right titles

pdfs = driver.find_elements_by_css_selector(".title a")
print(type(pdfs))
for my_pdf in pdfs:
    print(my_pdf.get_attribute('innerHTML'))

<class 'list'>
                                dfhgadr asfdgsd
                                sDF SDF 
                                SDFSDF sdf
                                SDF DSfdf
                                SDFD sdf
                                SDFSdf fdf
                                zewr4
                                sHSDFG 
                                RSZJRT
                                dhff fgf

score 0 · Accepted Answer · answered Jul 10 '21 at 16:15

0

Instead of pdfs[[i]].click() You may try as follows:

 i.click()

answered Jul 10 '21 at 16:15

Md. Fazlul Hoque

15,806
5
12
32

Hi. ´AttributeError: 'int' object has no attribute 'click'´ – id345678 Jul 10 '21 at 16:19
Try to use as: str(i).click() – Md. Fazlul Hoque Jul 10 '21 at 16:21
AttributeError: 'str' object has no attribute 'click' – id345678 Jul 10 '21 at 16:23
Instead of clic() , try to use Send_keys(Keys.ENTER) – Md. Fazlul Hoque Jul 10 '21 at 16:29
AttributeError: 'int' object has no attribute 'Send_keys' – id345678 Jul 10 '21 at 16:33
You may try a little bit change in for loop: for i in pdfs: – Md. Fazlul Hoque Jul 10 '21 at 16:37
that worked somewhat, thank you. It opened a pdf and then threw AttributeError: 'WebElement' object has no attribute 'WebDriverWait' – id345678 Jul 10 '21 at 16:41
Because sleep and WebDriverWait mixed are not allowed but you main problem has solved.Thanks – Md. Fazlul Hoque Jul 10 '21 at 16:44
I figured that out, thanks. Now it thows this: StaleElementReferenceException: Message: stale element reference: element is not attached to the page document. At `i.click()` – id345678 Jul 10 '21 at 16:54
1

Because element may remove from DOM.Please take help fron here:https://stackoverflow.com/questions/18225997/stale-element-reference-element-is-not-attached-to-the-page-document – Md. Fazlul Hoque Jul 10 '21 at 16:58

Scraping loop on list with next page using Selenium

1 Answers1