Python Selenium Explicit wait and send keys with xpath

Question

I am working on speeding up Selenium web scraping by replacing implicitly_wait to WebDriverWait with send_keys and click. I am a little confused on how to achieve that.

This is my code for inplicitly_wait:

def ncd_web_scraping(df):
    df['new_column'] = 'Not_sure'
    url = 'url'
    for i in df.index:
        driver = webdriver.Chrome()
        driver.implicitly_wait(5)
        driver.get(url)
        name = driver.find_element_by_xpath('//*[@id="person"]')
        name.send_keys(df.loc[i, 'Name'])
        state = driver.find_element_by_xpath('//*[@id="state"]')
        state.send_keys(df.loc[i, 'State'])
        botton = driver.find_element_by_xpath('/html/body/div[2]/form/button')
        botton.click()

        soup = BeautifulSoup(driver.page_source, 'html.parser')
        if soup.find('h5'):
            df.loc[i, 'new_column'] = 'Yes'
        else:
            df.loc[i, 'new_column'] = 'No'


    return df

Can anyone help me with how to webDriveWait to rewrite the code?

I appreciate your help.

Once the button clicked required HTML.Please post the HTML or share url? — KunduK, Jan 13 '20 at 15:59

undetected Selenium · Answer 1 · 2020-01-13T16:11:30.067

Replacing implicitwait with WebDriverWait your effective code block will be:

def ncd_web_scraping(df):
    df['new_column'] = 'Not_sure'
    url = 'url'
    for i in df.index:
        driver = webdriver.Chrome()
        driver.get(url)
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='person']"))).send_keys(df.loc[i, 'Name'])
        driver.find_element_by_xpath('//*[@id="state"]').send_keys(df.loc[i, 'State'])
        driver.find_element_by_xpath('/html/body/div[2]/form/button').click()
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        if soup.find('h5'):
            df.loc[i, 'new_column'] = 'Yes'
        else:
            df.loc[i, 'new_column'] = 'No'
    return df

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python Selenium Explicit wait and send keys with xpath

1 Answers1