Selenium: How to determine if WebdriverWait is really the problem?

Question

related to: ElementClickInterceptedException: Message: element click intercepted Element is not clickable error clicking a radio button using Selenium and Python and Debugging "Element is not clickable at point" error

I want to download every 10 pdfs from every page but it stops after 110 files at page 11 giving me this error:

ElementClickInterceptedException: Message: element click intercepted: Element <a href="/dsaf/sdgf/sdg.pdf" target="_blank" title="blabla bla">...</a> is not clickable at point (498, 9). Other element would receive the click: <div id="service-nav" class="header-right">...</div>

I searched and found that i would need to make the webdriver wait. I applied it already but i need to also get it to work on the following line of code i.click()

for j in range(295):
    pdfs = driver.find_elements_by_css_selector(".title a")
    if j < 10:
        pdfs = driver.find_elements_by_css_selector(".title a")
    for i in pdfs:
        time.sleep(3)
        i.click()
        time.sleep(3)
    if j < 10:
        driver.execute_script("arguments[0].click();", WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".page-next button"))))
    else:
        continue
break

When i replace i.click() with i.send_keys(Keys.ENTER) or

WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.CSS_SELECTOR, i))).click()

it doesent work as intended. Either it repeatedly download always the same pdf or downloads file 110-120 repreatedly or throws error: invalid argument: 'value' must be a string. How can i apply the webdriverwait thing to the i.click() instance?

here is the innerHTML of page 1

<head>
    <meta charset="utf-8">
    <meta http-equiv="x-ua-compatible" content="IE=edge">
    <meta name="generator" content="defah">

    <link rel="shortcut icon" href="/dfh.png" type="image/png">
    <script src="/tsartj.js" type="text/javascript"></script>
    <script>
        var enableP = 'False';
    </script>
        
    
    <link rel="alternate" type="application/rss+xml" title="RSS" href="http://afha.rss">
    

    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="robots" content="index, nofollow">
    <meta name="author" content="asgd">
    <meta name="description" content="sjdfg">
    <meta name="geo.region" content="sjsrt">
    <link rel="stylesheet" type="text/css" href="fdndfd" media="all">
    <link rel="stylesheet" type="text/css" href="fdndfd" media="all">
    <script src="vfdndfd" type="text/javascript"></script>
    <script src="fdndfd" type="text/javascript"></script>
    <script src="/fdndfd" type="text/javascript"></script>
    <script src="/fdndfd" type="text/javascript"></script>

    <title>fdndfd
    </title>
    <meta property="og:title" content="fdndfd">
    <meta property="og:type" content="Page">
    <meta property="og:url" content="http://pfdndfd">
    <meta property="og:site_name" content="fdndfd">
    <meta property="og:image" content="fdndfd">
    <script src="//fdndfd" type="text/javascript"></script><link id="rsmod_Styles" type="text/css" rel="stylesheet" href="//fdndfd">
    <link rel="canonical" href="http://fdndfd">
</head>
<body class="live script-on" data-rsevent-id="rs_780106">
    <ul id="skipnav" title="navigation">
        <li><a href="#main-nav">fdndfd</a></li>
        <li><a href="#main">fdndfd</a></li>
        <li><a href="#footer">fdndfdh</a></li>
        <li><a href="#service-nav">fdndfdn</a></li>
    </ul>

    <header id="header">
        <div class="container">
        
            <nav aria-label="fdndfd" id="main-nav" class="main-nav" aria-expanded="false">
                <button class="menu-button">
                    fdndfd
                </button>
                <div id="navigation" class="fdndfd">
                    <div class="container">
                        <nav class="service-nav only-mobil" aria-label="fdndfd">
    <ul>
        <li class="readspeaker-button" style="display: none;"><a href="https://fdndfd" title="fdndfd" rel="nofollow" onclick="readpage(this.href, 'readspeaker1'); return false;" target="_blank">fdndfd</a></li>
        <li><a href="fdndfd" title="fdndfd" target="_blank">fdndfd</a></li>
        <li><a href="fdndfd" title="fdndfd" target="_blank">fdndfd</a></li>
    </ul>
</nav>
                        <nav aria-label=" navigation">
                            <ul class="nav-points level-0" aria-label="fdndfd" role="menu">
                                <li tabindex="0" role="menuitem" aria-haspopup="false">
                                    <span tabindex="-1"><a href="https://fdndfd" title="fdndfd" target="_blank">fdndfd</a></span>
                                </li>
                                <li tabindex="0" role="menuitem" aria-haspopup="false">
                                    <span tabindex="-1"><a href="/fdndfd">fdndfd</a></span>
                                </li>

                                <li tabindex="0" role="menuitem" aria-haspopup="false">
                                    <span tabindex="-1"><a href="/fdndfd">fdndfd</a></span>
                                </li>
                                 <li tabindex="0" role="menuitem" aria-haspopup="false">
                                    <span tabindex="-1"><a href="/fdndfd">fdndfd</a></span>
                                </li>
                                
                            </ul>
                        </nav>
                    </div>
                </div>
            </nav>
            <div class="logo">
                <a href="https://fdndfd" title="fdndfd"><img src="/fdndfd" width="320" height="80" alt="fdndfd"></a>
            </div>
            
            <div id="service-nav" class="header-right">
                <nav class="service-nav no-mobil" aria-label="fdndfd">
                    <ul>        
                        <li class="font-size"><button tabindex="0" id="font-size-button" title="fdndfd" aria-expanded="false">fdndfd</button>
                            <ul>
                                <li class="font-size-1"><a href="#" title="Nfdndfd">A</a></li>
                            </ul>
                        </li>
                        <li class="readspeaker-button" style="display: none;"><a href="https://fdndfd" title="fdndfd" rel="nofollow" onclick="readpage(this.href, 'readspeaker1'); return false;" target="_blank">Sfdndfd</a></li>
                        <li><a href="https://fdndfd" title="fdndfd" target="_blank">fdndfd</a></li>
                        <li><a href="https://fdndfd" title="fdndfd" target="_blank">fdndfd</a></li>
                        <li><a href="/fdndfd" title="fdndfd">fdndfd</a></li>
                    </ul>
                </nav>
                <nav class="meta-menu no-cart" aria-label="Nfdndfde">
                    

                <div id="fdndfd" data-for="live" class="livefdndfd">
                    <a id="fdndfd" href="https://fdndfd" title="fdndfd" target="_blank">fdndfd</a>
                    
                    <div class="wrap-fix-live">
                        <div class="bg-grey">
                            <div class="container">
                                <a href="" class="icon-close" title="fdndfd"><span>fdndfd</span></a>
                                <div class="responsive-iframe">
                                </div>
                                <a id="fdndfd" class="open-live" rel="noopener" target="_blank" href="https://fdndfd"><span class="icon-live">fdndfd</span>
                                </a>
                            </div>
                        </div>
                    </div>

                </div>

                <!--div data-for="service" class="fdndfdlink btn">
                    <a href="https:/fdndfd" title="fdndfd" target="_blank">fdndfd</a>
                </div-->

                <!--button data-for="search" class="searchfdndfd" aria-expanded="false">
                    Suche
                </button>
                <div class="wrap-fix-search">
                    <div class="bg-grey modal-padding">

                    <form action="/fdndfd" method="post" onsubmit="return freeSearchTop();" id="tx_tltsearch" role="search" class="container container-search">
                        <label for="Search">
                            <input placeholder="fdndfd" class="search-box" id="topFreeSearch" name="SearchWords" type="text"/>
                            

<input type="hidden" value="1" name="SearchType" />
<input type="hidden" value="" name="fdndfd" />
<input type="hidden" value="" name="fdndddfd" />

                        </label>

                        <span class="icon-arrow" title="fdndfd">
                            <span tabindex="-1" class="icon-arrow-text">fdndfd</span>
                            <input class="search-submit" type="submit" value="fdndfd" />
                        </span>

                        <button tabindex="0" class="icon-close" title="fdndfd">
                            <span>fdndfd</span>
                        </button>
                    </form>

                    </div>
                </div-->
            </nav>
        </div>
    </div>
    </header>
    
    <main id="main">

    <div class="container include-in-search">
        <div id="keyvisual" class="keyvisual noprint">
            <img src="/fdndfd.png" width="1395" height="488" class="img-responsive" alt="">
            <div class="box-white clickable" onclick="location.href='/fdndfd';return false" title="fdndfd">
                <h1>
                    fdndfd<br>fdndfd
                </h1>
            </div>
        </div>
        <div id="keyvisualprintonly" class="keyvisual printonly">
            <h1>
                fdndfd
            </h1>
        </div>
        <div class="clearfix"></div>
    </div>

  

    <div class="container breadcrumb noprint">
        <div class="mainnavi tabs transformer-tabs rs_preserve" role="tablist">
    <button class="tabtoggle"><span class="icon-plus"><span class="icon-plus-text">fdndfd</span></span></button>
    <button role="tab" aria-controls="tab-1" aria-selected="false" onclick="location.href='http://fdndfd';return false;" id="tabheader-1" tabindex="-1">fdndfd</button>
    <button role="tab" aria-controls="tab-2" aria-selected="false" onclick="location.href='http://fdndfd';return false;" id="tabheader-2" tabindex="-1">fdndfd</button>
    <button role="tab" aria-controls="tab-3" aria-selected="true" onclick="location.href='http://fdndfd';return false;" id="tabheader-3" tabindex="-1">fdndfd</button>
</div>
        <!--ul id="breadcrumb" aria-label="fdndfd">
            
            <li><a href="/fdndfd" title="fdndfd">fdndfd</a><span class="separator"></span><span class="breadcrumb-selected">fdndfd</span></li>
        </ul-->        

        <div class="share-container">
            <button data-for="fdndfd" aria-label="fdndfd" class="share icon-fdndfd" aria-expanded="false">fdndfd</button>
            <div class="share-links">
                <a href="http://www.facebook.com/fdndfd" title="pfdndfd" target="_blank"><span class="icon-facebook">fdndfd</span></a>
                <a href="https://twitter.com/intent/fdndfd" title="pfdndfd" target="_blank"><span class="icon-twitter">fdndfd</span></a>
                <a href="https://api.whatsapp.fdndfd" title="fdndfd" class="only-mobil" target="_blank"><span class="icon-whatsapp">fdndfd</span></a>
            </div>
        </div>
    </div>
    <div id="readspeaker1" class="rs_skip rs_preserve"></div>
    
     <noscript>
      <div class="error-message">
      <br /><br /><br />
        fdndfd
      <br /><br /><br />
      </div>
    </noscript>
     <script type="text/javascript">
      //<![CDATA[
         if (!navigator.cookieEnabled) {
             document.write('<div class="error-message"><br /><br /><br />Cookies<br /><br /><br /></div>');
         }
      //]]>
     </script>

    
    <script type="text/javascript">
        //<![CDATA[
            $(document).ready(function () {
                $(".readspeaker-button").hide();
                $('#keyvisual').addClass('noprint');
                $('.breadcrumb').addClass('noprint');
            });
        //]]>
    </script>

    
<div class="container inner-container tlt_search printcontent">
    <h2 class="screenreader">fdndfd   
        
            Page 1
        
    </h2>
  
    

        <div class="page-head-height">
            <div class="pd_resultcount">
                
                    fdndfd
                1 - 10 von 2945 

               <span class="noprint">&nbsp;&nbsp;<a href="#" onclick="printStyle('fdndfd');return false">fdndfd</a>  &nbsp;&nbsp;</span>
                <br> <br>
                 

                
                            <a href="/fdndfd?param=true" id="search-refine" title="fdndfd">fdndfd</a>
                       
            </div>
        </div>
         <div class="tlt_search_results">
             <ul class="sourceGroup" aria-label="fdndfd">
        
        
            <li class="row tlt_search_result">
                <div class="col-1 docImage" data-index="index">
                     <a href="/fdndfd.pdf" title="fdndfd (436 KB)">
                        
                        <img src="/fdndfd.png" alt="fdndfd (436 KB)">
                     </a> 
                </div>
                <div class="col-11">
                    <div class="row title">
                        <div class="col-12">
                            <a href="/fdndfd.pdf" target="_blank" title="fdndfd">
                                fdndfd
                            </a>
                        </div>
                    </div>
                    <div class="row resultinfo">
                        <div class="col-12">
                            <div class="row">
                                <div>
                                    fdndfd
                                </div>
                            </div> 

                            
                                <div class="row">
                                    <div>
                                        fdndfd
                                    </div>
                                </div>
                             
                
                            <div class="row">
                                <div>
                                    fdndfd
                                    
                                    <span class="processlink">
                                        <a href="/Pfdndfd1" title="fdndfd">
                                            fdndfd
                                        </a>
                                    </span>
                                    
                                </div>
                                  
                            </div>

                            
                                
                        </div>
                    </div>
                </div>
            </li>
                          
            <li class="row tlt_search_result">
            .
            .
            .
            </li>

            <li class="row tlt_search_result">
            .
            .
            .
            </li>
             
            <li class="row tlt_search_result">
            .
            .
            .
            </li>

            <li class="row tlt_search_result">
            .
            .
            .
            </li>
             
            <li class="row tlt_search_result">
            .
            .
            .
            </li>

            <li class="row tlt_search_result">
            .
            .
            .
            </li>
             
            <li class="row tlt_search_result">
            .
            .
            .
            </li>

            <li class="row tlt_search_result">
            .
            .
            .
            </li>
             
            <li class="row tlt_search_result">
                <div class="col-1 docImage" data-index="index">
                     <a href="/fdndfd.pdf" title="fdndfd">
                        
                        <img src="/fdndfd.png" alt="fdndfd (435 KB)">
                     </a> 
                </div>
                <div class="col-11">
                    <div class="row title">
                        <div class="col-12">
                            <a href="/fdndfd.pdf" target="_blank" title="fdndfd">
                                fdndfd
                            </a>
                        </div>
                    </div>
                    <div class="row resultinfo">
                        <div class="col-12">
                            <div class="row">
                                <div>
                                    fdndfd
                                </div>
                            </div> 

                            
                                <div class="row">
                                    <div>
                                        fdndfd
                                    </div>
                                </div>
                             
                
                            <div class="row">
                                <div>
                                    fdndfd
                                    
                                    <span class="processlink">
                                        <a href="/fdndfd" title="fdndfd">
                                            fdndfd
                                        </a>
                                    </span>
                                    
                                </div>
                                  
                            </div>

                            
                                
                        </div>
                    </div>
                </div>
            </li>
             
           </ul>
        <div class="tlt_search_footer"><form class="has-validation-callback"><nav class="pagination" aria-label="fdndfd"><ul><li class="page-first disable"><a role="link" aria-disabled="true" aria-label="fdndfd"><span class="icon-arrow"></span><span class="icon-arrow"></span></a></li><li class="page-prev disable"><a role="link" aria-disabled="true" aria-label="fdndfd"><span class="icon-arrow"></span></a></li><li class="tx-pagebrowse-pages pages"><ol><li class="tx-pagebrowse-current"> 1</li><li class="tx-pagebrowse-page"><button type="submit" aria-label="fdndfd" title="fdndfd" onclick="location.href='/fdndfd';return false"> 2</button></li><li class="tx-pagebrowse-page"><button type="submit" aria-label="Page 3" title="Go to page 3" onclick="location.href='/fdndfd';return false"> 3</button></li></ol></li><li class="page-next"><button type="submit" aria-disabled="false" aria-label="fdndfd" onclick="location.href='/fdndfd';return false;"><span class="icon-arrow"></span></button></li><li class="page-last"><button type="submit" aria-disabled="false" aria-label="fdndfd" onclick="location.href='/fdndfd';return false;"><span class="icon-arrow"></span><span class="icon-arrow"></span></button></li></ul></nav></form></div>
       
             <form class="has-validation-callback"><button class="btn noprint" type="submit" aria-label="fdndfd" name="Back" onclick="location.replace('/fdndfd');return false">fdndfd</button></form>
          </div>
    

    
</div>

    
    <div class="clearfix"></div>
    </main>

    <footer id="footer">
        <div class="container">
    <div class="row">
        <div class="col1">
            <h2>fdndfd<button class="icon-plus" data-for="fdndfd" aria-label="fdndfd"></button></h2>
            <ul id="kontakt">
                <li><a href="https://fdndfd" title="fdndfd" target="_blank">fdndfd</a></li>
            </ul>
        </div>
        <div class="col2">
            <h2>fdndfd<button class="icon-plus" data-for="sprache" aria-label="pfdndfd"></button></h2>
            <ul id="sprache">
                <li><a href="https://pfdndfd" title="pfdndfd" target="_blank">pfdndfd</a></li>
                <li><a href="https://pfdndfd" title="pfdndfd" target="_blank">pfdndfd</a></li>
            </ul>
        </div>
    </div>
</div>
<div class="container footer-social-media">
    <div class="row links-social-media">
        <ul>
            <li><a href="https://www.facebook.com/pfdndfd" target="_blank" title="pfdndfd"><span class="icon-facebook">Facebook</span></a></li>
        </ul>
    </div>
</div>
        <img src="/footer.png" width="1600" height="1005" class="bg-footer" alt="">
        <div class="clearfix"></div>
    </footer>

    <div class="grt-cookie grt-cookie-active" style="color: rgb(36, 36, 36); background: rgb(255, 255, 255);">
        <div class="grt-cookies-msg">
            <p class="h2 grt-cookies-header">Cookies</p>
            <p>Accept CookiesLink: <a href="https://pfdndfd" target="_blank">pfdndfd</a>.</p>
        </div>
        <span class="grt-cookie-button" style="background: rgb(0, 108, 183); color: rgb(255, 255, 255);">Ok</span>
    </div>
    <script src="/pfdndfd.js" type="text/javascript"></script>
    <!-- Default texts and parameters -->
    

<div id="hidden-param">
    <!-- URLs -->
    <span id="url-update-descriptors">/pfdndfd</span>
    <span id="url-descriptor-references">/pfdndfd</span>
    <span id="url-reload-speaker-persons">/pfdndfd</span>

    <!-- MESSAGES -->
    <span id="msg-unknown-error">pfdndfd</span>
    <span id="msg-reload">pfdndfd.</span>
    <span id="msg-unknown-error-page-reload">Epfdndfd</span>
    <span id="msg-empty-keywords">pfdndfd.</span>
    <span id="msg-empty-subjectareas">pfdndfd.</span>
    <span id="msg-wrong-date-format">pfdndfd</span>

    <span id="msg-no-solution-set">pfdndfd</span>
    <span id="msg-captcha-no-number">pfdndfd</span>
    
</div>



</body>

Can you share a link to the web site? I'm quite sure the problem is definitely not with the webdriver wait itself... — Prophet, Jul 12 '21 at 13:21
thanks for your comment. Unfortunately i cannot share. Would html help? — id345678, Jul 12 '21 at 14:07
Maybe. But not just of that element. Preferably of the entire page. — Prophet, Jul 12 '21 at 14:13
i edited it accordingly and replaced search result 2 to 9 with `

Prophet · Accepted Answer · 2021-07-13T09:15:43.393

1

I guess the issue is:
There are 295 pages with 10 links on each of them so as you can click next button each time in order to go to the next page.
The next page button is initially inside the visual view of the screen, however after 11 pages it becomes out of the screen or covered by some other element.
I think you should move to the next page button to bring it into the visible screen area. To do so you can instead of

driver.execute_script("arguments[0].click();", WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".page-next button"))))

try using this:

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time


wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)

next_btn = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".page-next button")))
time.sleep(0.2)
actions.move_to_element(next_btn).click().perform()

UPD
I see a logical problem with your code.
You just do nothing for j >= 10. So you are entering endless loop on j==10.
Also it's unclear why you sure you will have 295 pages?
Why not simply iterate until next page button is existing?
Please instead of this:

for j in range(295):
    pdfs = driver.find_elements_by_css_selector(".title a")
    if j < 10:
        pdfs = driver.find_elements_by_css_selector(".title a")
    for i in pdfs:
        time.sleep(3)
        i.click()
        time.sleep(3)
    if j < 10:
        driver.execute_script("arguments[0].click();", WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".page-next button"))))
    else:
        continue
break

Use something like this:

while True:
    pdfs = driver.find_elements_by_css_selector(".title a")
    for i in pdfs:
        time.sleep(3)
        i.click()
        time.sleep(3)
    next_btn = driver.find_elements_by_css_selector(".page-next button")
    if next_btn:
        actions.move_to_element(next_btn[0]).click().perform()
        time.sleep(3)
    else:
        break

edited Jul 13 '21 at 09:15

answered Jul 12 '21 at 16:37

Prophet

32,350
22
54
79

It works on the first 11 pages and correctly downloads the first 110 files before it enter an endless loop on page 11 again, without hitting the next-page button. Here are my chrome webdriver options: `prefs = {"profile.default_content_settings.popups": 0, "download.default_directory": r"C:\Users\...", "directory_upgrade": True, "plugins.always_open_pdf_externally": True}` `driver.maximize_window()`. – id345678 Jul 13 '21 at 07:34
Need to see the actual web page + dev tools – Prophet Jul 13 '21 at 07:41
1

I have updated the code. See if now it works correct – Prophet Jul 13 '21 at 09:16
the first 10 files work and then: `WebDriverException: Message: chrome not reachable` which in this situation makes no sense to me as my chrome and chromedriver versions are compatible, i checked that by `driver.capabilities` – id345678 Jul 13 '21 at 09:57
1

That's strange.. As mentioned here https://stackoverflow.com/questions/49205782/selenium-common-exceptions-webdriverexception-message-chrome-not-reachable-err try setting this: – Prophet Jul 13 '21 at 10:06
1

chrome_options.add_argument('--no-sandbox') chrome = webdriver.Chrome('/usr/local/bin/chromedriver', chrome_options=chrome_options) – Prophet Jul 13 '21 at 10:07
No Sandbox doesent help, i get the same error `options = webdriver.ChromeOptions() options.add_argument("--no-sandbox") prefs = {"profile.default_content_settings.popups": 0, "download.default_directory": r"C:\Users\..", "directory_upgrade": True, "plugins.always_open_pdf_externally": True} options.add_experimental_option("prefs",prefs) driver = webdriver.Chrome(executable_path=DRIVER_PATH, options=options)` – id345678 Jul 13 '21 at 10:19
If so currently I have no idea what can be the problem. I didn't face with this. – Prophet Jul 13 '21 at 10:52
1

rebooting turned out to be helpful although i did `tearDown(self): self.driver.close()`. Now i get `NameError: name 'actions' is not defined`. Therefore i placed `actions = ActionChains(driver)` before your `actions.move_to_element` command and until now it works even for page 12 and onwards. Thank you so much :) – id345678 Jul 13 '21 at 11:40
I'm happy I could assist you! – Prophet Jul 13 '21 at 11:53

Selenium: How to determine if WebdriverWait is really the problem?

1 Answers1