2

I am new to python and trying to do some webscraping but have some real issues. May be you can help me out.

HTML:

<input autocomplete="off" type="search" name="search-search-field" placeholder="150k companies worldwide" data-cy-id="search-search-field" class="sc-dnqmqq grpFhe" value="">

The first part of my code looks as follows and works good without having any issues:

driver.get("https:")
login = driver.find_element_by_xpath(email_xpath).send_keys(email)
login = driver.find_element_by_xpath(pwd_xpath).send_keys(pwd)
login = driver.find_element_by_xpath(continue_xpath)
login.click()
time.sleep(10)

email and pwd are variables including my login details. As I said that part works pretty fine.

The issues I have are with the following code line:

search = driver.find_element_by_xpath('/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input')

As a result I get this following error:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input"}

I tried and tried but could not solve the problem. I would appreciate it very much, if anyone could help me out. Thank you!

Economist Learning Python
  • 306
  • 1
  • 3
  • 17

3 Answers3

2

To locate the search field you can use either of the following Locator Strategies:

  • Using css_selector:

    search = driver.find_element_by_css_selector("input[name='search-search-field'][data-cy-id='search-search-field']")

  • Using xpath:

    search = driver.find_element_by_xpath("//input[@name='search-search-field' and @data-cy-id='search-search-field']")

Ideally, to locate the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='search-search-field'][data-cy-id='search-search-field']")))

  • Using XPATH:

    search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='search-search-field' and @data-cy-id='search-search-field']")))

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

References

You can find a couple of relevant discussions on NoSuchElementException in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Hi DebanjanB, ty so much, but unfortuanelty neither using css_selector nor using your xpath example worked for me. I still get errors like: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"input[name='search-overlay-search-field'][data-cy-id='search-overlay-search-field']"} or if I use WebDriverWait: search = Web....) File "D:\...", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: – Economist Learning Python Jan 07 '21 at 09:25
  • @Numpy This is the actual reason why we need the text based relevant HTML of the element so that we can test our solution before publishing it. – undetected Selenium Jan 07 '21 at 09:28
  • Ty, you mean this here? – Economist Learning Python Jan 07 '21 at 09:56
  • @Numpy Checkout the updated answer and let me know the status. – undetected Selenium Jan 07 '21 at 10:04
  • Ty, unfortunately doesn't work, here the error code: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[@name='search-search-field' and @data-cy-id='search-search-field']"} – Economist Learning Python Jan 07 '21 at 10:08
  • How about inducing _WebDriverWait_? – undetected Selenium Jan 07 '21 at 10:09
  • Unfortunately still getting an error: Traceback (most recent call last): File "D:\...py", line 39, in search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='search-search-field' and @data-cy-id='search-search-field']"))) File "D:\...\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: – Economist Learning Python Jan 07 '21 at 10:19
  • @Numpy and what about the Css w/out WebDriverWait? – undetected Selenium Jan 07 '21 at 10:20
  • Unfortunately same problem: Traceback (most recent call last): File "D:\....py", line 39, in search = driver.find_element_by_css_selector("input[name='search-search-field'][data-cy-id='search-search-field']") ... selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"input[name='search-search-field'][data-cy-id='search-search-field']"} – Economist Learning Python Jan 07 '21 at 10:28
  • 1
    This is so weird. Since I used another code line maximizing the browser window your suggestion using the xpath method, it's running without any errors. I use this following code: search_xpath = "//input[@name='search-search-field' and @data-cy-id='search-search-field']" driver.maximize_window() search = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, search_xpath))).send_keys("test") – Economist Learning Python Jan 07 '21 at 16:27
  • @Numpy This is a part of your test and possibly you mustn't even think about executing your tests without **maximizing the browser window**, that should be done by _default_. Of coarse instead of `presence_of_element_located()` you need to use `element_to_be_clickable()` – undetected Selenium Jan 07 '21 at 17:10
  • 1
    Ty for your support DebanjanB, element_to_be_clickable also works fine, but only if I used maximize_window() before. – Economist Learning Python Jan 07 '21 at 17:49
  • @Numpy While executing your tests `maximize_window()` isn't **optional** but **mandatory** – undetected Selenium Jan 07 '21 at 17:50
  • Thank you, may be a last question. Can you tell me why it is mandatory having the browser window maximized? Ty in advance! – Economist Learning Python Jan 07 '21 at 19:11
  • @Numpy Check the discussion [How to execute tests with selenium webdriver while browser is minimized](https://stackoverflow.com/questions/52504503/how-to-execute-tests-with-selenium-webdriver-while-browser-is-minimized/52515176#52515176) – undetected Selenium Jan 07 '21 at 19:14
  • 1
    Great, ty for your support! – Economist Learning Python Jan 07 '21 at 20:11
0

The following Xpath will work and is much simpler.

/html/body/div[1]//fieldset/input
James
  • 387
  • 1
  • 11
  • Ty James, worked for me! Can you tell me how you managed to get the xpath. The xpath I used, I just used the inspection option of Google Chrome and copied the full xpath. – Economist Learning Python Jan 07 '21 at 09:17
  • using the chrome developer tools I noticed that the element you are looking for is the first instance of a fieldset element. I simplified you path to get the input in the first fieldset. Which could actually be simplified more to `//fieldset/input` – James Jan 07 '21 at 22:27
  • Great, ty James! – Economist Learning Python Jan 07 '21 at 22:37
0

do not use absolute xpath or css , always use relative as it is more stable

Absolute (Full) xpath will be depended on parent and so if parent changes the locator will fail to find the element

in xpath and css the locator can be used in the form:

//tagname[@attributename="attriubutevalue"]   - Xpath
tagname[attributename="attriubutevalue"]      - CSS

so you can use any attribute , type, name , id ,class, what ever attribute there in your element eg:

//input[@type="search"]  - xpath
input[type="search"]   - css


search = driver.find_element_by_xpath('//input[@type="search"]')

Try wait:

WebDriverWait(driver,15).until(EC.presence_of_element_located((By.XPATH, '//input[@type="search"]')))

enter image description here

PDHide
  • 18,113
  • 2
  • 31
  • 46
  • Ty PDHide, do you have a specific program code that I can test? – Economist Learning Python Jan 07 '21 at 09:29
  • @Numpy search = driver.find_element_by_xpath('//input[@type="search"]') – PDHide Jan 07 '21 at 09:57
  • Ty, but unfortunately still getting following error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[@type="search"]"} – Economist Learning Python Jan 07 '21 at 10:00
  • @Numpy what happens when you use wait ? WebDriverWait(driver,15).until(EC.presence_of_element_located((By.XPATH, '//input[@type="search"]'))) , there is no reason it shouldn't work – PDHide Jan 07 '21 at 10:07
  • Unfortunatelly still an error returned: WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, '//input[@type="search"]'))) File "...venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: – Economist Learning Python Jan 07 '21 at 10:15
  • This is so weird. Since I used another code line maximizing the browser window your suggestion using the xpath method is running without any errors. I use this following code: search_xpath = "//input[@name='search-search-field' and @data-cy-id='search-search-field']" driver.maximize_window() search = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, search_xpath))).send_keys("test") – Economist Learning Python Jan 07 '21 at 16:29