How to loop through all pages of search results, select href for each search result, scrape data using python and selenium

Question

I am scraping a website, where i need to select a value from dropdown. Then i need to click on search button. Which then populates a table with multiple pages of search results. For each result i want to click on it, which redirects to another page, extract data from that. Then come back and do the same for other search results.

select = Select(browser.find_element_by_id("dropdown id here"))
options = select.options
for index in range(0,len(options)):
    select.select_by_index(index)
    browser.find_element_by_id("checkbox").click()
    time.sleep(5)
    browser.find_element_by_id("click on search").click()
    elems = browser.find_elements_by_xpath("need to enter all href for search results")
    time.sleep(5)
    for elem in elems:
        #Need to enter each search result's href scrape within data and get back to search results again##
        elem.get_attribute("href")
        elem.click()
        
        browser.find_element_by_id("for going back to search page again").click()

I get this error when i'm trying to iterate

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (Session info: chrome=87.0.4280.88)

Have you checked the error code meanings in the [selenium docs](https://www.seleniumhq.org/exceptions/stale_element_reference.jsp)? — William Baker Morrison, Dec 09 '20 at 16:24

Gaurav Khurana · Answer 1 · 2020-12-10T10:00:46.477

1

This problem comes when the element is not able to find out as it has become stale

Try to bring the find elemenet logic inside for loop, that should solve your problem.

    elems = browser.find_elements_by_xpath((xpath)[i])
    length = elems.size();

Run your logic with number

for (int i=0;i<length; i++)
    #Need to enter each search result's href scrape within data and get back to search results again##

    elems = browser.find_elements_by_xpath("need to enter all href for search results"[i])

    elem.get_attribute("href")
    elem.click()
    time.sleep(1)

you can read more about it here https://stackoverflow.com/a/12967602/2986279

edited Dec 10 '20 at 10:00

answered Dec 09 '20 at 13:29

Gaurav Khurana

3,423
2
29
38

When I'm declaring "i" for elems like you did its not able to find the element – Pratyush Dash Dec 09 '20 at 15:22
do like ((xpath)[i]) so your xpath is dynamic everytime,, and also if you can please accept this as answer if it works for you – Gaurav Khurana Dec 10 '20 at 10:01

score 0 · Answer 2 · answered Dec 09 '20 at 13:16

0

If you are opening new windows, you will likely want to switch focus between newly opened windows/tabs and the original.

Switch to newly opened window.

windows = browser.window_handles
browser.switch_to.window(windows[-1])

Close newly opened window after finding what you need on the new page, then switch back to the previous window.

browser.close()
windows = browser.window_handles
browser.switch_to.window(windows[-1])

answered Dec 09 '20 at 13:16

Jason Cook

1,236
9
12

When i click on the search result, It doesn't open up a new window..It redirects on the same page.. – Pratyush Dash Dec 09 '20 at 13:23
Unable to paste a picture here of the screenshots – Pratyush Dash Dec 09 '20 at 13:26

How to loop through all pages of search results, select href for each search result, scrape data using python and selenium

2 Answers2