Selenium unable to locate though using expected conditions

Question

Selenium is unable to locate the element by its xpath even though I'm using expected conditions to wait 3 seconds for element visibility. I'm trying to interact with the search bar, send a string, and press enter. My code is below:

def search_bar(self, search: str, xpath: str):
        """
        Looks for the search bar on the website and sends the search string

        Input: 
            search (str)
                the string to be search on
            xpath (str)
                the value of the xpath where the search is located
        

        """
        #search_bar = self.driver.find_element(by=By.XPATH, value=f'{xpath}')
        search_bar = self.wait.until(EC.element_to_be_clickable((By.XPATH, xpath )))
        search_bar.click()
        search_bar.send_keys(search)
        search_bar.send_keys(Keys.RETURN)

I think that it waits for the element to be clickable, then I can click it and send my search to it, (the website I am using is https://uk.trustpilot.com/).

scraper = Scraper()
scraper.load_webpage('https://uk.trustpilot.com/')
scraper.click_on_cookie('//*[@id="onetrust-accept-btn-handler"]')
scraper.search_bar('travel insurance', '//*[@id="heroSearchContainer"]')

Definition of my scraper class:

class Scraper:
    def __init__(self):
        """
       Defines the webdriver for the scraper class
        
        """
        _edge_driver = os.path.join(os.getcwd(), 'msedgedriver.exe') #path to the edge driver
        options = Edgeoptions()
        options.add_argument('start-maximized')
        self.driver = webdriver.Edge(executable_path=_edge_driver, options= options)
        self.wait = WebDriverWait(self.driver,3)

Finally by the error message which I am receiving is the following:

selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable

Can the problem be that i haven't provided the full xpath as opposed to the shortened id i provided — Darman, Jan 19 '23 at 20:49
sort of weird to use the method name as a variable name in the method. I would fix that first. You may also want to add a sleep after clicking the cookie prompt. The login field may not be interactable until after the cookie is saved by your browser successfully. (it doesn't take much time, but it does take some) — pcalkins, Jan 19 '23 at 21:06
@pcalkins Yeah i need to change it, i just put it there as a easier way to see what i'm doing, that makes sense actually, i'll try having a go with it, is there also a better way to do it, say for example waiting till the document state is ready or would that not work? — Darman, Jan 19 '23 at 21:11
don't check document/readiness state... there'd be timing issues with that. Just do a webdriverwait for an expected condition that appears when it's ok to continue. It's hard to say what that might be... could be a "save cookie preference" text, or the invisibility of an overlay (which could cause timing issues too) It'll depend on the site. One solution would be to catch this specific exception and try again. (re-call same method from exception... you can function-ize it to recall and set a counter in the class as a sanity count.) — pcalkins, Jan 19 '23 at 21:17
I make the sanity count = 2x the webdriverwait timeout. (webdriverwait polls every 1/2 second by default.) I use it for stale element exceptions which can happen if the DOM is still loading and you're getting an array of elements that's still populating. — pcalkins, Jan 19 '23 at 21:28
make sure your click_on_cookie method is also using the element_to_be_clickable expected condition. (you don't show that method here) Did you try the answer below? — pcalkins, Jan 19 '23 at 23:26

score 0 · Accepted Answer · answered Jan 19 '23 at 21:22

0

To send a character sequence to the search field you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Code Block:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver.get('https://uk.trustpilot.com/')
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='query']"))).send_keys("Darman")

Browser Snapshot:

Darman

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Jan 19 '23 at 21:22

undetected Selenium

183,867
41
278
352

1

Hey yeah that seemed to have worked, sorry i already implemented it, another problem i was facing was the wait times, a lot of it just seemed to run way to quickly so i had to add a few explicit waits – Darman Jan 20 '23 at 20:07
Can i ask you a further question that i'm having a problem with, so i navigated to this part of the site https://uk.trustpilot.com/categories, i had noticed that every catogory is under a h2 tag which i would like to extract to give the user more automation on which catogory they would like, i tried catogories = self.driver.find_elements(By.TAG_NAME,value ="h2") – Darman Jan 20 '23 at 20:30
@Darman Sounds like we would need more details for this new usecase. – undetected Selenium Jan 20 '23 at 20:32
No problem i'll upload a new question and then attach the link to the next comment – Darman Jan 20 '23 at 20:36
https://stackoverflow.com/questions/75189164/selenium-how-to-locate-all-elements-with-the-tag-h2 – Darman Jan 20 '23 at 20:43

Selenium unable to locate though using expected conditions

1 Answers1