previous web crawler doesn't recognize element id

Question

I am new to the web crawling task. Previously I tried the following simple crawler, and it worked well. Recently I come back to the code and tried to do more on crawler, however the browser.find_element_by_id("lst-ib") does not work and I receive the error that says

' no such element: Unable to locate element: {"method":"css selector","selector":"[id="lst-ib"]"} (Session info: chrome=84.0.4147.89) '

To solve my problem, I tried to find xpath of input text box for google page from inspect. Is it always like that? does the id or css selector that we define for crawler change regularly and we should update the code?

  from selenium import webdriver

  url = "https://www.google.com"

  browser = webdriver.Chrome(executable_path = "chromedriver")
  browser.get(url)
  

  #inputElement = browser.find_element_by_id("lst-ib")

  # I replace the xpath with previous id
  inputElement = 
  browser.find_element_by_xpath("/html/body/div/div[2]/form/div[2]/div[1]/div[1]/div/div[2]/input")

  inputElement.send_keys("my input search text")
  inputElement.submit()

  browser.quit()

try `inputElement = browser.find_element_by_xpath('//input[@title="Search"]')` — JaSON, Jul 30 '20 at 08:01
@JaSON Still I receive this error: Unable to locate element: {"method":"xpath","selector":"//input[@title="Search"]"} (Session info: chrome=84.0.4147.89). Am I doing sth wrong? — Mahsa, Jul 30 '20 at 08:07
Does [this discussion](https://stackoverflow.com/questions/55713414/elementnotvisibleexceptionelement-not-interactable-error-locating-google-sear/55715154#55715154) help you? — undetected Selenium, Jul 30 '20 at 08:21
@Mahsa , then try to apply [ExplicitWait](https://selenium-python.readthedocs.io/waits.html#explicit-waits) — JaSON, Jul 30 '20 at 08:24

SeleniumUser · Accepted Answer · 2020-07-30T09:05:26.547

1

try below xpath :

inputElement = 
  browser.find_element_by_xpath("//body[@id='gsr']/div[@id='viewport']/div[@id='searchform']/form[@id='tsf']/div/div/div/div/div/input[1]")

inputElement.send_keys("my input search text")

Your solution:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(executable_path=r"path of chrome driver")

wait = WebDriverWait(driver, 10)
driver.get("https://www.google.com")

inputElement = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "/html/body/div/div[2]/form/div[2]/div[1]/div[1]/div/div[2]/input")))
inputElement.send_keys("my input search text")

Output :

edited Jul 30 '20 at 09:05

answered Jul 30 '20 at 08:19

SeleniumUser

4,065
2
7
30

it does work, do you know why the previous one inputElement = browser.find_element_by_id("lst-ib") does not work any more? how you generate this xpath? I used copy xpath from inspect and I got : /html/body/div/div[2]/form/div[2]/div[1]/div[1]/div/div[2]/input which is different that yours – Mahsa Jul 30 '20 at 08:38
I have added your example too, So you need to use WebDriverWait to avoid snchronization issue. In your solution you are using absolute path during run time web driver searching your element through DOM and because you are using absolute xpath its taking time to locate your element. Try to avoid use of aabsolute xpath .Also its always good to use WebDriverWait to avoid synchronizzation issues – SeleniumUser Jul 30 '20 at 09:04
@Mahsa : Would you kind enough to accept answer and hit upvote button from your end if your issue is resolved. – SeleniumUser Jul 30 '20 at 09:07

previous web crawler doesn't recognize element id

1 Answers1