ElementNotVisibleException: Message: element not visible - Python3 Selenium

Question

I have been tasked with writing a parser to click a href link, that looks like a button, on a website and I am having some issues.

Here's the html: https://pastebin.com/HDKLXpdJ

Here's the source html: https://pastebin.com/PgT91kJs

Python code:

browser = webdriver.Chrome()
...
try:

    element = WebDriverWait(browser, 20).until(
        EC.presence_of_element_located((By.ID, "reply-panel-reveal-btn")))

finally:
      elem = browser.find_element_by_xpath("//A[@id='reply-panel-reveal-btn']").click()

I am getting this error.

selenium.common.exceptions.ElementNotVisibleException: Message: element not visible

I have tried switching between ChromeDriver and GeckoDriver(FF), but I'm getting the same error, over and over again. I even tried waiting for 10 secs to load, same results.

Full error text:

File "C:/Users/DEM/PycharmProjects/Test/Scrape.py", line 46, in <module> elem = browser.find_element_by_xpath("//A[@id='reply-panel-reveal-btn']").click()
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 77, in click self._execute(Command.CLICK_ELEMENT)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 493, in _execute return self._parent.execute(command, params)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 256, in execute self.error_handler.check_response(response)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 6.1.7601 SP1 x86_64)

Video link on how it should work :

https://streamable.com/e1uvm

Edit:

Problem solved, check @JeffC answer.

The correct code :

browser = webdriver.Chrome()
...
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "(//a[@id='reply-panel-reveal-btn'])[2]")));
element.click()

The problem :

I was waiting for the presence of the element. Presence doesn't mean that the element is visible or clickable, it just means the element is in DOM. Also, I was waiting for the first element, which happens to be invisible. I needed to locate the second element and just wait for it to be clicked.

sometimes, dynamic content is not generated if you don't scroll. You can simulate this behavior in JavaScript — rak007, Oct 13 '17 at 12:27
share the link, i think there is iframe involved. anyways share the link here if you want help — Stack, Oct 13 '17 at 12:39
@Stack here's the link - https://www.gumtree.com/p/audi/-reduced-audi-a3-se-tdi-sportback-s-line-2005-with-rotor-alloys-18-/1269429904 — Zygimantas Drukteinis, Oct 13 '17 at 12:39
By the way, you need to log in here to be able to reveal the phone number. I can quickly create an account for you . — Zygimantas Drukteinis, Oct 13 '17 at 12:40
@ZygimantasDrukteinis if you need to scroll down to generate content, i often use this one liner : browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") — rak007, Oct 13 '17 at 12:43
@rak007 thank you for a nice suggestion. This could be useful in some cases, but in this case, it didn't work. — Zygimantas Drukteinis, Oct 13 '17 at 12:47

score 1 · Accepted Answer · answered Oct 13 '17 at 13:11

1

There are a couple issues.

You are waiting for presence of the element. Presence just means that the element is in the DOM, not that it's visible or clickable. If you are going to wait and click an element, wait for it to be clickable. If you are going to wait to send_keys() or get the text from an element, wait for it to be visible. The are some uses for presence but I don't use it often. Having said that...
There are two elements that match your locator, id=reply-panel-reveal-btn. The first one that match just happens to be invisible. With XPath we can create a locator that finds the second element, wait for it to be clickable, and then click it.
```
element = WebDriverWait(driver, 20).until(
    EC.element_to_be_clickable((By.XPATH, "(//a[@id='reply-panel-reveal-btn'])[2]")));
element.click()
```

answered Oct 13 '17 at 13:11

JeffC

22,180
5
32
55

1

Basically, all I needed to do is just to wait for the second element to be clickable and then click the second element that isn't invisible, right? Thank you @JeffC! Now I can finally finish my project. I'm not joking, you just made my day. – Zygimantas Drukteinis Oct 13 '17 at 13:24
1

One thing that I use all the time is Chrome's dev tools to locate elements on the page. It's a quick way to verify that your locators are working and you are finding the element(s) you expect. `$$()` is for CSS selectors, e.g. `$$("span.someClass")`. `$x()` is for XPaths, e.g. `$x("//a[@id='someId']")`. More info [here](https://developers.google.com/web/tools/chrome-devtools/console/expressions) – JeffC Oct 13 '17 at 13:45
If you run `$$("#reply-panel-reveal-btn")` on your page, you will see that it comes back with 2 elements. If you expand the return and hover each one, you will see that the second one is the one that's visible on the page. It's a HUGE time saver and I use it all the time. – JeffC Oct 13 '17 at 13:47
You're right. By typing $$("#reply-panel-reveal-btn") in dev tools console, it returned two elements. This will be so useful when dealing with problems like this one, makes the whole situation much clearer. Thanks again for helping me out! – Zygimantas Drukteinis Oct 13 '17 at 15:20

score 0 · Answer 2 · answered Oct 13 '17 at 12:25

0

could you try to click on

//span[@class='icn-phone icn-quaternary']

or

//div[@class='clearfix']

answered Oct 13 '17 at 12:25

Gaurang Shah

11,764
9
74
137

Thank you for your suggestion. I tried both paths and I'm still getting the same error. – Zygimantas Drukteinis Oct 13 '17 at 12:31
how about using explicit wait for visibility? – Gaurang Shah Oct 13 '17 at 12:47
@Guarang Shah will try that now – Zygimantas Drukteinis Oct 13 '17 at 12:51
@Guarag Shah tried to time.sleep(20) to sleep for 20 secs, didn't work. – Zygimantas Drukteinis Oct 13 '17 at 12:52

ElementNotVisibleException: Message: element not visible - Python3 Selenium

2 Answers2

Linked