Web scraping with selenium leave button not clickable

Question

I think it's similar to this issue: Button not clickable in Selenium Chrome window

I need to search information about a list of logements in an automatic way. I'm trying selenium but it cannot click the button to perform the search. Maybe it's some protection from the website ?

So far my script is:

import time
import string
import itertools
from selenium import webdriver

chromedriver_location = "/home/dell/Downloads/chromedriver"

driver = webdriver.Chrome(chromedriver_location)
driver.get('https://www.idealista.com/valoraciones-inmuebles')

time.sleep(2)

test = '7559431DG0075N0001EY'
initial_button = '//*[@id="vendorlead"]/div[2]/a'
search_field = '//*[@id="search-by-reference"]'
valoration = '//*[@id="vendorlead"]/button'
driver.find_element_by_xpath(initial_button).click()
driver.find_element_by_xpath(search_field).send_keys(test)
driver.find_element_by_xpath(valoration).click()

When I click the button manually or through script in Selenium Chrome it shows the errors below on console:

Thanks in advance

The website cannot find the address you gave. You should give a valid address so that you can click to the link. — M.Soyturk, Jul 09 '20 at 15:23
@M.Soyturk I don't think the problem is that. The only thing my script can't do is click the final button to see the results — joaopfg, Jul 09 '20 at 15:38

score 1 · Accepted Answer · answered Jul 10 '20 at 12:43

You're being blocked. (doing a 3rd answer as the other ones are very valid in the debugging approach for automation)

Run your tests and let it fail to press that final button - but don't close chrome. In that chromedriver instance, select the language option in the top right and select English.

You'll go through a LOT of i-am-not-a-robot captcha's. Complete them all until you hit the English site. (You won't be able to automate these - they're designed to stop automation.)

Close chrome driver.

Rerun your script as is.

...

The above worked for me. First time i ran the below it failed with your same error. I did the above, ran the same code without change and it worked.

import time
import string
import itertools
import selenium
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://www.idealista.com/valoraciones-inmuebles')

test = '7559431DG0075N0001EY'

#Objects
initial_button = '//*[@id="vendorlead"]/div[2]/a'
#search_field = '//*[@id="search-by-reference"]'
search_field ='//*[@id="vendorlead"]//input'
valoration = '//*[@id="vendorlead"]/button'

#Actions
time.sleep(5)
driver.find_element_by_xpath(initial_button).click()
time.sleep(5)
driver.find_element_by_xpath(search_field).send_keys(test)
time.sleep(5)
driver.find_element_by_xpath(valoration).click()
time.sleep(5)

...Just please don't use that time.sleeps in a real script. i just needed to rule out synchronisation issues.

python 3.8, chrome 83

This would probably be why the c# worked for me - i hadn't done enough to qualify as a robot on the sit :-) — RichEdwards, Jul 10 '20 at 13:07

score 0 · Answer 2 · answered Jul 09 '20 at 17:24

0

Do you need some sync between the send keys and the click search?

I had a quick look at the site and when I manually enter the test data the site runs some scripts. If you try and press search with no data, nothing happens.

Try a time.sleep between the send keys and the click as a quick test. If that works, use a Web driver wait to synchronise dynamically.

answered Jul 09 '20 at 17:24

RichEdwards

3,423
2
6
22

I don't think the problem is in sync between send keys and click search because I cannot click the button in selenium drive. Maybe there is some protection against running it on selenium drive? Could you try to run it, please ? – joaopfg Jul 09 '20 at 18:13
Running it as you have it - i get the same problem. I also have the same issue in c# - so it's the site not the bindings... however, looked a little more the problem might not the click - the problem is the send keys. If you enter the data manually and then get selenium to press the button then it works – RichEdwards Jul 10 '20 at 07:39
It's the send keys object - i'll post a new reply in a sec – RichEdwards Jul 10 '20 at 07:44

RichEdwards · Answer 3 · 2020-07-10T09:00:10.660

0

As mentioned in the comments on my other answer, i tried your code and got the same result. I also tried c# so it's not the selenium bindings.

I tried and few things manually and looks like the the underlying issue is the sendkeys wasn't setting the data for the form.

I couldn't find the object you had - so i just used devtools to find a new one and that worked.

If you change:

search_field = '//*[@id="search-by-reference"]'

to be:

search_field ='//*[@id="vendorlead"]//input'

That worked for me in c#.

I know you're doing python but this is what i did in c# to make it work:

   [Test]
        public void stackoverflow()
        {

            var driver = new ChromeDriver();
            driver.Url = "https://www.idealista.com/valoraciones-inmuebles";
            driver.Manage().Window.Maximize();

            var test = "7559431DG0075N0001EY";
            string initial_button = "//*[@id='vendorlead']/div[2]/a";
            string search_field = "//*[@id='vendorlead']//input";
            string valoration = "//*[@id='vendorlead']/button";
            driver.FindElement(By.XPath(initial_button)).Click();
            driver.FindElement(By.XPath(search_field)).SendKeys(test);
           driver.FindElement(By.XPath(valoration)).Click();


        }

edited Jul 10 '20 at 09:00

answered Jul 10 '20 at 07:51

RichEdwards

3,423
2
6
22

In python it still does not work. Could you try it, please? – joaopfg Jul 10 '20 at 08:45
Do you get any error or any feedback when you run it? do you know how to run tests in debug mode so you can step through each action? - i just edited the answer to remove a surplus " and ; from the "to be" part.... Can you double check that these objects are found? – RichEdwards Jul 10 '20 at 09:01
Thanks for help. I updated the original post with the error displayed in the console when I try to click the button. Can you understand that? – joaopfg Jul 10 '20 at 09:58
Are you running the script at full speed or debugging? - can you step through every line to ensure the calls are complete before the next? - you're error is 403 forbidden and an unhanded promise (i.e. an async call) - Going back to my original post you might need to just do it slower between every action (with proper sync) This site updates the dom with async calls when you do an action. If you're interacting before/while it's doing these updates it's going to cause unpredictable behavior. Selenium is for web - but this is script powered (like react or angular), that needs more libraries. – RichEdwards Jul 10 '20 at 10:11
I'm just debugging it by trying to pass a value. But even if I just open this page with Selenium Chrome it doesn't allow to click manually in the button. And a weird thing is that it happens only with this button – joaopfg Jul 10 '20 at 10:39
By the way, how could I run your C# script? – joaopfg Jul 10 '20 at 10:47
It works for me so the button/page/site/doing-automation isn't the problem. It's python-selenium or your script/process. If you want to run the c# flavor, you'll need to install visual studio to compile it - it's massively off topic and more than i can type in comments. I'll try find time to install python and try things again and get back to you when i can. – RichEdwards Jul 10 '20 at 10:59
One more thing to try - launch chromedriver, go to the URL but do nothing, top right corner of the page - change the language to English. Complete the captcha (the "not a robot") - change languat back to espaniol, close the chromedriver and try again - running it slowly! one step at a time - with a 5 second wait between each step! (even between nagivate to url and first click) . Your IP/useragent combo might be blocked. – RichEdwards Jul 10 '20 at 11:21
I saw that I can compile C# with 'mcs' and run it with 'mono' on Linux. Could you give me the full code so that I can try here, please? – joaopfg Jul 10 '20 at 12:12

Web scraping with selenium leave button not clickable

3 Answers3