Why search retrieve no result? with selenium

Question

I have been trying to scrape this site and sending in the search phrase cause error.

AttributeError: 'NoneType' object has no attribute 'send_keys'

I have searched StackOverflow and the error usually is resolved when you redefine the web element again which I did but still getting the same error.

I do not want to directly go to the search link like https://www.couriermail.com.au/search-results?q=Anthony+Albanese instead I want to click on the search icon and after the input field is open send in the search phrase.

Here is what I have tried

            WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
            search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()

            search.send_keys("Anthony Albanese")
            time.sleep(10)
            search.send_keys(Keys.ENTER)

I have tried simulating the typing-like behavior but it results in other errors stackoverflow help

This is the HTML of the page that I am referring to The HTML of the search icon so my question is what I am doing wrong? I have tried redefining the web element or imitating the typing-like behavior but I am not successful in getting the URLs of articles

score 1 · Answer 1 · answered Aug 25 '22 at 07:48

This code line

search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()

Assigns nothing i.e. 'NoneType' object into the search object because

driver.find_element(by=By.XPATH,value='/html/body/nav/form/button')

returns a WebElement but it is not assigned to search object on the left side since you are applying a click() method on that returned WebElement while the click() method returns nothing.
You can fix this your code simply like the following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button')
search.click()
search.send_keys("Anthony Albanese")
time.sleep(10)
search.send_keys(Keys.ENTER)

You can also improve your code by removing the hardcoded delay of time.sleep(10) - Expected Conditions explicit waits will resolve most of these issues by significantly improving the text run time.
Also this XPath locator '/html/body/nav/form/button' can and should be improved.

your answer will send only the last term "albanese" not the whole phrase so it result into this link https://www.couriermail.com.au/search-results?q=Albanese and if I increase the sleep time it simply don't search and stay on the home page — snoozy, Aug 25 '22 at 07:56
Well, maybe we need to add some delay between the `search.click()` and the `search.send_keys("Anthony Albanese")` ? — Prophet, Aug 25 '22 at 08:01
I tried that but it will result into same link with no articles — snoozy, Aug 25 '22 at 08:28

Anand Gautam · Accepted Answer · 2022-08-25T08:30:56.183

Yet another answer, but with relative xpaths instead of absolute. Also, instead of simulating ENTER key, I have used the clicking of search button.

driver.get("https://www.couriermail.com.au/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@class='navigation']//button[@name='Search']"))).click()
input_text = driver.find_element(by=By.XPATH, value="//form[contains(@class, 'navigation_search')]//input")
input_text.send_keys("Anthony Albanese")
print(input_text.get_attribute('value'))
time.sleep(2)
driver.find_element(By.XPATH, "//*[@class='navigation']//button[@name='Search']").click()
time.sleep(20)
# WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[@class='storyblock_title']")))
all_articles = driver.find_elements(By.XPATH, "//*[@class='storyblock_title']")
article_headers = [article.text for article in all_articles]
print(article_headers)
print(driver.current_url)
assert driver.current_url == "https://www.couriermail.com.au/search-results?q=Anthony+Albanese"
driver.quit()

Output:

   Anthony Albanese
['Albo saves Cutters’ PNG fixture after boycott threat', 'Sport diplomacy: Why a QRL game is so important to the PM', 'Albo’s cheeky dig at ScoMo during Midnight Oil event', 'How Albo amassed huge property portfolio', '‘I can’t solve everything’: PM’s big admission', 'PM admits: ‘Can’t solve everything’', "Anthony Albanese 'emulated Bob Hawke'", "People 'cheered like idiots' at Anthony Albanese skolling beer", "'Hold my beer': Anthony Albanese 'trying to be as popular as Hawke'", 'I want to look at ‘how issues could be improved in the future’: Anthony Albanese', 'PM Anthony Albanese Signals Review Of Scott Morrison’s Conduct Is On The Horizon 23/08/2022', 'Standing ovation as Albo necks beer', 'Albo chugs beer at Gang of Youths concert', 'Aussies being left behind by the PM', 'Prime Minister’s big jobs promise', 'Huge push for foreign workers in Australia', 'Can we afford another mouth to feed?', 'SA projects under review after former PM’s power snatch', 'Albanese hints at ScoMo inquiry', '‘Never again’: Albo flags huge action']
https://www.couriermail.com.au/search-results?q=Anthony+Albanese

Process finished with exit code 0

P.S. print and sleep statements are for easy visual. They are optional and could be omitted. If needed by explicit_wait like webdriverwait could be used.

thanks for the answer. I will try it but did you get the articles too? thanks for showing me how to make the XPath relative btw. — snoozy, Aug 25 '22 at 08:08
I tried it but got an empty page but with no data or articles — snoozy, Aug 25 '22 at 08:28
@snoozy I modified my answer. But one thing is it doesn't work everytime, I don't know, perhaps, due to bot detection. It failed numerous times, and passed only once, i.e., the articles loaded only once in about 10 attempts I made. Maybe you need to use a proxy or stealth browser — Anand Gautam, Aug 25 '22 at 08:32
thanks a lot. Yeah that's the problem I have been using fake agent though but still, it causes the same issue and I bought the subscription too — snoozy, Aug 25 '22 at 08:37
so I do try with stealth browser too but it's the same. but I accepted your answer because at least I was able to reach the searched linked — snoozy, Aug 25 '22 at 10:12

score -1 · Answer 3 · answered Aug 25 '22 at 07:46

Your search should be of type WebElement but instead you define search as call of click method. Try to replace

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()
search.send_keys("Anthony Albanese")

with

search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button")))
search.click()
search.send_keys("Anthony Albanese")

Why search retrieve no result? with selenium

3 Answers3