python selenium wait for page to load

Question

I've written a script that gets data from a page, but sometimes the page takes time to load and so when it pull the html into a soup object sometimes It pulls nothing as the page still needs to finish.

I wrote the following code for waiting the page to finish.

def scrape_page(url):
     browser.get(url)    
     try:
        WebDriverWait(browser, 10).until(EC.presence_of_element_located(browser.find_element_by_id ("selection-box")))
        #Extract Source Code 
        html = browser.page_source;
        soup = BeautifulSoup(html)

It works

But I'm getting the following error when I call the function;

TypeError: find_element() argument after * must be a sequence, not WebElement

WKPlus · Accepted Answer · 2017-05-11T06:50:56.013

13

I think you should use presence_of_element_located like this:

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )

as described in the manual.

edited May 11 '17 at 06:50

answered May 20 '15 at 03:45

WKPlus

6,955
2
35
53

I tried this originallly, , but get the error global name By is not defined. Only thing that is different is that Im using Chrome Driver instead of firefox – Grant McKinnon May 20 '15 at 04:03
http://selenium-python.readthedocs.org/en/latest/api.html?highlight=#module-selenium.webdriver.common.by – sobolevn May 20 '15 at 04:24
4

@GrantMcKinnonG import `By` before use it: `from selenium.webdriver.common.by import By` – WKPlus May 20 '15 at 06:29
@GrantMcKinnon- You have to import as from selenium.webdriver.common.by import By – Learner Jul 15 '15 at 11:41
1

The link in this answer has died. RIP link. – Andrew May 09 '17 at 18:12

score 4 · Answer 2 · answered Mar 13 '22 at 08:09

After 6 years from this question, all the solutions I looked for was not good for my situation, I found the solution in other programming language and I did implementation for Python, so this is the best way to wait a page to fully load.

WebDriverWait(self.driver, self.SERVER_TIMEOUT).until(
        lambda wd: self.driver.execute_script("return document.readyState") == 'complete',
        "Page taking too long to load"
    )

score 1 · Answer 3 · edited Apr 20 '21 at 14:38

I apply this func to every WebElement I need to use.

from selenium import webdriver

def FindElem(Driver: webdriver, XPath: str, Timeout: int = 300):
    while Timeout > 0:
        try:
            return Driver.find_element_by_xpath(XPath)
        except: # if element isn't already loaded or doesn't exist
            time.sleep(1)
            Timeout -= 1
    raise RuntimeError(f"Page loading timeout") # or whatever the hell you want

Usage:

Driver = webdriver.Firefox()
webdriver.get("http://somewhere.com/somepage.html")
MyWebElement = FindElem(Driver, "//input[@name='email']") # raise exception if timeout

python selenium wait for page to load

3 Answers3

Linked