Python Selenium issue with submit button, not working consistently

Question

I am trying to build a web scraper for redfin to pull the redfin estimate. I have a function that does this and sometimes it works and sometimes it does not work. I noticed that the reason it does not work is because of the submit function. Sometimes the chrome page wont press the search (submit) button and go to the property page.

I am not sure how to fix this issue and make it more consistent.

Here is my code:

from selenium import webdriver
from selenium.webdriver.remote import webelement
import pandas as pd
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.redfin.com/')
time.sleep(3)

def get_redfin_estimate(address):
    search_box = driver.find_element_by_name('searchInputBox')
    search_box.send_keys(address)
    search_box.submit()
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    try:
        price1 = soup.find('div', {'class', 'avm'}).div.text
        return(price1)
    except AttributeError:
        try:
            time.sleep(10)
            price2 = soup.find('span',class_='avmLabel').find_next('span', class_='value').text
            return(price2)
        except:
            return('N/A')


print(get_redfin_estimate('687 Catalina Laguna Beach, CA 92651'))
print(get_redfin_estimate('693 Bluebird Canyon Drive, Laguna Beach, CA 92651'))



driver.quit()

Have you tried not using `.submit()` and actually clicking on the magnifying glass element to submit the search? — JeffC, Jun 19 '19 at 21:29
@JeffC I am totally new to this stuff, if you have an answer please provide one. — Wolfy, Jun 19 '19 at 21:30
Delete the `search_box.submit()` line, find the element by the CSS selector `button[title='Search']`, and click it. — JeffC, Jun 19 '19 at 21:32
@JeffC I am sure how to do that can you provide the line I need to replace for search_box.submit() please — Wolfy, Jun 19 '19 at 21:34

undetected Selenium · Answer 1 · 2019-06-19T21:54:31.817

1

The reason submit button is not working consistently because the desired element is a JavaScript enabled element and the Locator Strategy you have used doesn't identifies the search box with placeholder as City, Address, School, Agent, ZIP uniquely and identifies 3 elements.

To send a character sequence to the desired field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

def get_redfin_estimate(address):
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.searchInputNode input.search-input-box#search-box-input"))).send_keys(address)
    driver.find_element_by_css_selector("div.searchInputNode button.inline-block.SearchButton.clickable").click()

Using XPATH:

def get_redfin_estimate(address):
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='searchInputNode']//input[@class='search-input-box' and @id='search-box-input']"))).send_keys(address)
    driver.find_element_by_xpath("//div[@class='searchInputNode']//button[@class='inline-block SearchButton clickable float-right']").click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Jun 19 '19 at 21:54

answered Jun 19 '19 at 21:07

undetected Selenium

183,867
41
278
352

Thank you, could you update the code I have in your answer? – Wolfy Jun 19 '19 at 21:14
@Wolfy I didn't get you. How can I help you? – undetected Selenium Jun 19 '19 at 21:16
I understand your answer, but could you update to include my code with the appropriate changes that you suggested? – Wolfy Jun 19 '19 at 21:17
Does that make sense? I just want to make sure my code is perfectly corrected given the changes you suggested... – Wolfy Jun 19 '19 at 21:19
@Wolfy Check out the answer update and let me know if you have got the expected answer. – undetected Selenium Jun 19 '19 at 21:21
Sorry I will try to be more clear. Can you copy and paste my code from the question and make the appropriate changes that you suggested in my code? I hope that is more clear – Wolfy Jun 19 '19 at 21:22
@Wolfy No, the answers job is not to copy/paste the entirety of your code, it is to answer your question. The code here should be enough for you to see what the differences are and move that into your code on your own. You are already getting free advice... why can't you move the suggested code into your own by yourself? – JeffC Jun 19 '19 at 21:24
@JeffC Given the answers he provided with the changes that I made I still get the same problem. I.e. sometimes it works and sometimes it does not... – Wolfy Jun 19 '19 at 21:26
That's because his answer isn't really any different than what you already had. Your locator actually finds three elements but the first is the one you wanted so that doesn't matter. He just (over) specified the first element with a locator that uniquely finds the first element... but it doesn't change what actually happens. – JeffC Jun 19 '19 at 21:28
@Wolfy Checkout the updated answer and let me know the status. – undetected Selenium Jun 19 '19 at 21:55

JeffC · Accepted Answer · 2019-06-19T21:42:17.173

1

It may be an issue with using .submit() on that element. One alternative is to just click on the magnifying glass to initiate the search.

def get_redfin_estimate(address):
    driver.find_element_by_name('searchInputBox').send_keys(address)
    driver.find_element_by_css_selector("button[title='Search']").click()
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    ... and so on

edited Jun 19 '19 at 21:42

answered Jun 19 '19 at 21:40

JeffC

22,180
5
32
55

driver.find_element_by_css_selector('button[title='Search']').click() ^ SyntaxError: invalid syntax – Wolfy Jun 19 '19 at 21:41
Sorry, fixed typo – JeffC Jun 19 '19 at 21:42
You didn't get the final update. I hit save too quickly and only changed one typo. You really need to spend some time reading some python tutorials so you can fix these basic items on your own. It was just a case of mismatched quotes. – JeffC Jun 19 '19 at 21:44
Yea I know I was asked to build this at work and I normally do research in ML and use C++ to write algorithms so I am totally new to this – Wolfy Jun 19 '19 at 21:46

Python Selenium issue with submit button, not working consistently

2 Answers2