Based on the comment conversation, I would recommend both using an XPATH search (instead of Name or Id) and waiting for elements to be clickable or loaded. When web-driving or web-scraping, pages may intentionally or accidentally load slowly and this can cause issues if you have pauses or waits either hard coded or non-existent. This snippet of code should allow you to search Google using Selenium and Chromedriver (you can modify the driver function to use Firefox or something else if you'd like):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import ElementNotVisibleException
from selenium.webdriver.chrome.options import Options
from time import sleep
def init_driver(drvr_path):
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
driver = webdriver.Chrome(drvr_path+'chromedriver.exe',chrome_options=chrome_options)
driver.wait = WebDriverWait(driver, 5)
return driver
def lookup(query, driver=None, drvr_path=''):
driver = None
if driver is None:
driver = init_driver(drvr_path)
driver.implicitly_wait(45) # Allow up to 45 Seconds for page to load
driver.get("http://www.google.com")
try:
box = driver.wait.until(EC.presence_of_element_located((By.XPATH, """//*[@id="lst-ib"]""")))
box.send_keys(query)
sleep(3) # Let you see the window open
button = driver.wait.until(EC.element_to_be_clickable((By.XPATH,"""//*[@id="sblsbb"]/button""")))
try:
button.click()
except ElementNotVisibleException, s:
print "Error Handled: "+str(s)
button = driver.wait.until(EC.element_to_be_clickable((By.XPATH,"""//*[@id="sblsbb"]/button""")))
try:
button.click()
except:
print "Could not search Google..."
return
resp=driver.page_source.encode('utf-8')
with open(query+'.html','wb') as f:
f.write(resp)
print 'Wrote the File...'
except:
print("Box or Button not found in google.com")
driver.quit()
For example, if your Chromedriver.exe file was located in your default Python path, you could do something like: lookup('Selenium Python XPATH Examples')
and it should download an HTML file of the Google Search results. If you already have a Driver initialized, you could of course pass that to it.
Hope this helps