How to go through the pages and extract the link from them with Slenium Python?

Question

I'm having problems capturing the link of the pages.

Below is the code, terminal output message and HTML image where the link is located.

classe = driver.find_elements(By. XPATH, "//*[@class='LinksShowcase_UrlContainer__kMj_n']/p")

for pages in range(2,6):
    driver.get("https://www.xxxx.com.br/painel/my-links?page="+ str(pages)+"&items=6&q=&startDate=&endDate=&popular=0")
    sleep(2)

    for i in classe:
        #pages += 1
        links.append(i.text)
        print(links)
        sleep(2)

terminal output message:
Traceback (most recent call last):
  File "c:\Users\felip\OneDrive\Área de Trabalho\Web\VS\Link Afiliado\teste.py", line 61, in <module>
    links.append(i.text)
  File "C:\Users\felip\OneDrive\Área de Trabalho\Web\Py\lib\site-packages\selenium\webdriver\remote\webelement.py", line 89, in text
    return self._execute(Command.GET_ELEMENT_TEXT)["value"]
  File "C:\Users\felip\OneDrive\Área de Trabalho\Web\Py\lib\site-packages\selenium\webdriver\remote\webelement.py", line 410, in _execute        
    return self._parent.execute(command, params)
  File "C:\Users\felip\OneDrive\Área de Trabalho\Web\Py\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 444, in execute
    self.error_handler.check_response(response)
  File "C:\Users\felip\OneDrive\Área de Trabalho\Web\Py\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 249, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document        
  (Session info: chrome=108.0.5359.125)

javascript technique can fix this, refer this for reason https://stackoverflow.com/questions/71881850/again-stale-element-reference-element-is-not-attached-to-the-page-document — Dickens A S, Dec 31 '22 at 02:34
Calling `driver.get()` loads a new page into the browser, and so the elements in `classe` become stale . — John Gordon, Dec 31 '22 at 02:36
I tried without success wait.until(EC.presence_of_element_located(links.append(i.text))) — Felipe, Dec 31 '22 at 02:53
try javascript within python https://stackoverflow.com/questions/7794087/running-javascript-in-selenium-using-python — Dickens A S, Dec 31 '22 at 07:12
AbiSaran it is a site where you need to register to access the URL that I need to extract — Felipe, Dec 31 '22 at 14:32
John Gordon you solved my problem, in view of what you said I simply changed the position of FOR, I will leave my wrong code in the post but I will post a print in this comment so you can understand what was done. https://prnt.sc/dkMHl3i9ee-4 https://prnt.sc/e63PuxVTu_tS — Felipe, Dec 31 '22 at 14:39

score 1 · Answer 1 · answered Dec 31 '22 at 07:38

Based on my comment with reference of to stackoverflow and also from my experience, any html element which rendered beyond the browser viewpoint which still going to appear based on user focus or scroll you can use javascript to get the element

from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://stackoverflow.com/questions/7794087/running-javascript-in-selenium-using-python") 
link = driver.execute_script("document.getElementById('LinksShowcase_UrlContainer__kMj_n')")

The output of the javascript is stored in variable link
This variable link could be None in python, in case if that element still not rendered by the browser so you can code like

if link != None:
    print("link rendered")
else:
    print("link not rendered")

`if link is not None:` better then. – Сергей Кох Dec 31 '22 at 08:38 — Сергей Кох, Dec 31 '22 at 08:38

How to go through the pages and extract the link from them with Slenium Python?

1 Answers1