Click not working for selenium in web scraping using python

Question

I am working on web scraping using selenium(python). There is chunk on code which is really long. So , i am using loops. When i am running the code lines separately, it is working fine but when i am using the loop, its not working. Here are the two errors:

WebDriverException: Message: unknown error: Element is not clickable at point (862, 13). Other element would receive the click: <div class="skycom_container">...</div>
(Session info: chrome=46.0.2490.80)
(Driver info: chromedriver=2.20.353145 (343b531d31eeb933ec778dbcf7081628a1396067),platform=Windows NT 6.1 SP1 x86_64)

WebDriverException: Message: unknown error: Element is not clickable at point (924, 786). Other element would receive the click: <div id="silentUIblocker" style="display: block;"></div>
(Session info: chrome=46.0.2490.80)
(Driver info: chromedriver=2.20.353145 (343b531d31eeb933ec778dbcf7081628a1396067),platform=Windows NT 6.1 SP1 x86_64)

Are these common or specific errors?? This is happening just before using the click() statement.

Here is my code:

from selenium import webdriver
import time
driver = webdriver.Chrome('C:\Users\name\Downloads\chromedriver_win32 (3)\chromedriver.exe')


driver.get('https://www.sky.com/shop/beta?s_tnt=87085:31:0')
driver.find_element_by_xpath('//*[@id="app"]/div/div/div[2]/div/div[5]/article/button/div[1]/div[2]/div/h2').click()
driver.find_element_by_xpath('//*[@id="app"]/div/div/div[2]/div/div[6]/section/div/div/div/div/div[1]/article/a').click()
driver.find_element_by_xpath('//*[@id="polaris"]/div/div/div/section/div/div[2]/a[2]').click()
driver.find_element_by_xpath('//*[@id="dsl-postcode"]').send_keys("E11 2LX")
driver.find_element_by_xpath('//*[@id="dsl-check-landline"]').click()
driver.find_element_by_xpath('//*[@id="dsl-addresses"]/option[2]').click()
driver.find_element_by_xpath('//*[@id="dsl-multiple-address-select"]').click()
driver.find_element_by_xpath('//*[@id="dsl-numberPortingNo"]').click()
driver.find_element_by_xpath('//*[@id="dsl-number-porting-submit"]').click()
driver.find_element_by_xpath('//*[@id="summaryBackLink"]').click()
driver.find_element_by_xpath('//*[@id="oneOffCostToolTip"]').click()
bb_pack = ["SKY_FIBRE_CAPPED", "BB_MAX"]
for i in bb_pack:
  driver.find_element_by_xpath('//*[@id="productButtonControls_%s"]/label' % i).click()
  bb_name1.append(driver.find_element_by_xpath('//*[@id="productButtonControls_%s"]/label' % i).text)
  pack = ["ANYTIME_EXTRA", "INTERNATIONAL_EXTRA"]
  for j in pack:
    driver.find_element_by_xpath('//*[@id="productButtonControls_ST_%s"]/label' % j).click()
    bb_name2.append(driver.find_element_by_xpath('//*[@id="productButtonControls_ST_%s"]/label' % j).text)
    #more details in this loop

Answer 1

Use fluent wait as below. It will wait till your element is not ready on page:-

WebElement waitsss(WebDriver driver, By elementIdentifier){
     Wait<WebDriver> wait =
                new FluentWait<WebDriver>(driver).withTimeout(60, TimeUnit.SECONDS)
                                                 .pollingEvery(1, TimeUnit.SECONDS)
                                             .ignoring(NoSuchElementException.class);

return wait.until(new Function<WebDriver, WebElement>()
        {
            public WebElement apply(WebDriver driver) {
                   return driver.findElement(elementIdentifier);
            }
            });
}

The wait should work for you. If still the problem exists then use JavascriptExecutor . It will operate directly through JS. It should work. I am giving an example to click any element using JavascriptExecutor

WebElement element = driver.findElement(By.id("gbqfd"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);

I just saw you are using python, in python, it should be something like below :-

driver.execute_script("arguments[0].click();", element)

The script should be like below :-

driver.execute_script("document.getElementsByClassName('skycom_container')[0].click()")
driver.execute_script("document.getElementById('silentUIblocker').click()")

Final code

    driver = webdriver.Chrome('C:\Users\name\Downloads\chromedriver_win32 (3)\chromedriver.exe')
    driver.implicitly_wait(30) # seconds
    driver.get('https://www.sky.com/shop/beta?s_tnt=87085:31:0')
    driver.find_element_by_xpath('//*[@id="app"]/div/div/div[2]/div/div[5]/article/button/div[1]/div[2]/div/h2').click()
    element=driver.find_element_by_xpath('//*[@id="app"]/div/div/div[2]/div/div[6]/section/div/div/div/div/div[1]/article/a') 
    driver.execute_script("arguments[0].click();", element)
    driver.find_element_by_xpath('//*[@id="polaris"]/div/div/div/section/div/div[2]/a[2]').click()
    driver.find_element_by_xpath('//*[@id="dsl-postcode"]').send_keys("E11 2LX")
    driver.find_element_by_xpath('//*[@id="dsl-check-landline"]').click()
    driver.find_element_by_xpath('//*[@id="dsl-addresses"]/option[2]').click()
    driver.find_element_by_xpath('//*[@id="dsl-multiple-address-select"]').click()
    driver.find_element_by_xpath('//*[@id="dsl-numberPortingNo"]').click()
    driver.find_element_by_xpath('//*[@id="dsl-number-porting-submit"]').click()
    driver.find_element_by_xpath('//*[@id="summaryBackLink"]').click()
    driver.find_element_by_xpath('//*[@id="oneOffCostToolTip"]').click()

Hope it will help you :)

Answer 2

Try to add below imports

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

and insert below line just before making click():

WebDriverWait(driver, 10).until(EC.invisibility_of_element_located((By.ID, "silentUIblocker")))

This should allow you wait until element that intercept the click disappeared