How to clear search area when using selenium for web scraping

Question

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.common.keys import Keys
import datetime
import pandas as pd
import requests


driver = webdriver.Chrome('./chromedriver')
driver.set_window_size(800,600)
driver.implicitly_wait(5)
driver.get('https://news.google.com/?hl=ko&gl=KR&ceid=KR%3Ako')
driver.implicitly_wait(2)

words = ['atom','mechanic','clock']

for word in words:

    if datetime.datetime.today()==0:
        word = word+'when:3d'
    else:
        word = word +'when:1d'

    elem = driver.find_element_by_xpath('//*[@id="gb"]/div[2]/div[2]/div/form/div[1]/div/div/div/div/div[1]/input[2]')
    elem.send_keys(Keys.CONTROL + 'a' + Keys.NULL, word.decode('utf-8'))

    elem.send_keys(Keys.RETURN)

Above is the entire script for web scraping the rest is working fine but this part, elem.send_keys(Keys.CONTROL + 'a' + Keys.NULL, word.decode('utf-8')), doesn't do anything. Any tips on how to fix this to clear search area for next words?

Answer 1

To clear the search area when using Selenium you have to induce WebDriverWait for theelement_to_be_clickable() and you can use the following Locator Strategy:

• Using XPATH:

  element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='gb']/div[2]/div[2]/div/form/div[1]/div/div/div/div/div[1]/input[2]")))
  element.click()
  element.clear()
  

• Note : You have to add the following imports:

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC