0

I write a python script. first, it visits this website. then click on the arrow on the right side and go to the new web page to collect some data. finally back to the previous page and do the same thing with next item.

Web page : https://register.fca.org.uk/s/search?q=capital&type=Companies

This is the code.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.wait import WebDriverWait
import time

url = 'https://register.fca.org.uk/s/search?q=capital&type=Companies'
service = Service('link to come driver')
service.start()
driver = webdriver.Remote(service.service_url)
driver.get(url)
time.sleep(12)


divs = driver.find_elements_by_xpath('//div[@class="result-card_main"]')
for d in divs:

   RN = ''
   companyName = ''
   companyName =   d.find_element_by_tag_name('h2').text
   RNData = d.find_element_by_xpath('.//div[@class="result-card_figure-offset"]').text
   RN = RNData.split(':')[1].strip()

   d.click()
   time.sleep(12)

   phoneNumber = ''
   phoneNumberData =  driver.find_elements_by_xpath('//*[@id="who-is-this-details-content"]/div[1]/div[2]/div[2]/div/div/div[2]')
   phoneNumber = phoneNumberData[0].text.split('\n')[1]

   print(RN)
   print(companyName)
   print(phoneNumber)

   driver.execute_script("history.back();")

it givesme this Error:

  selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

How can I solve this problem?

dfcsdf
  • 187
  • 1
  • 2
  • 8
  • When you navigate away and go back, the references become stale. You want to find a way to iterate those that doesn't use element references – pguardiario May 31 '21 at 02:44
  • how can I do that? – dfcsdf May 31 '21 at 03:24
  • 1
    Does this answer your question? [StaleElementReferenceException on Python Selenium](https://stackoverflow.com/questions/27003423/staleelementreferenceexception-on-python-selenium) – Just Mohit May 31 '21 at 04:03

1 Answers1

1

Here's a quick and dirty way to avoid that error, change your code like this:

url = 'https://register.fca.org.uk/s/search?q=capital&type=Companies'

driver.get(url)
time.sleep(12)


divs = driver.find_elements_by_xpath('//div[@class="result-card_main"]')
for i in range(len(divs)):
   time.sleep(4)
   d = driver.find_elements_by_xpath('//div[@class="result-card_main"]')
   RN = ''
   companyName = ''
   companyName = d[i].find_element_by_tag_name('h2').text
   RNData = d[i].find_element_by_xpath('.//div[@class="result-card_figure-offset"]').text
   RN = RNData.split(':')[1].strip()

   d[i].click()
   time.sleep(12)

   phoneNumber = ''
   phoneNumberData =  driver.find_elements_by_xpath('//*[@id="who-is-this-details-content"]/div[1]/div[2]/div[2]/div/div/div[2]')
   phoneNumber = phoneNumberData[0].text.split('\n')[1]

   print(RN)
   print(companyName)
   print(phoneNumber)

   driver.execute_script("window.history.go(-1)")
C. Peck
  • 3,641
  • 3
  • 19
  • 36
  • IndexError: list index out of range – dfcsdf May 31 '21 at 03:47
  • Sorry, I didn't test it before. Now I did. For some reason your navigation "back" didn't work for me so I found a different way. I also found a wait is needed before trying to identify `d` in the loop. I updated the code with what works for me. – C. Peck May 31 '21 at 04:01
  • Comment: using hard sleeps is brittle and takes too long. You should look into using [webDriverWait](https://selenium-python.readthedocs.io/waits.html) in your test(s). – C. Peck May 31 '21 at 04:04
  • Thanks. it works. I will definitely add webDriverWait to it. :) – dfcsdf May 31 '21 at 04:12
  • what kind of change I need to do in this code if I want to go next page and do the same thing again and again. – dfcsdf May 31 '21 at 05:15
  • I'm not sure exactly what you mean. If you're facing another issue, you can ask another question. – C. Peck May 31 '21 at 05:19
  • No issue with this code, this code collect data from page 1, so I try to reach page 2 using for loop & adding below code to it, – dfcsdf May 31 '21 at 05:48
  • bt = driver.find_element_by_xpath('//*[@id="-pagination-next-btn"]') bt[0].click() it it gives same list index out of range – dfcsdf May 31 '21 at 05:48