0

I'm trying to get an HTML text off a webpage using python selenium. But Selenium appears to be unable to locate the element. Not sure if I am doing anything wrong and is looking for some answers here.

Please see a part of my code below:

wait(driver, 10).until(EC.visibility_of_element_located((By.ID, 'current_address_name')))
full_address = driver.find_element_by_id('current_adress_name')
street_address = full_address.text.split('(S)')[0]
print('Street Address: ' + str(street_address))
all_street_address.append(street_address)

This is the error message (timeout). Supposedly because it cannot find the element:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Volumes/GoogleDrive/My Drive/PycharmProjects/googlestuff/gsheet_test_mod.py", line 121, in web_extraction
    wait(driver, 10).until(EC.visibility_of_element_located((By.ID, 'current_address_name')))
  File "/Users/cadellteng/googstuff/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

And this is a part of the HTML code:

<div id="current_address_name" style="display:none">4 Third Avenue (S)266576</div>

A little background about this thing though. This part of the element doesn't seem to be a visible element on the page because when I put my mouse over the element, it does not highlight any part of the page. Is it because Selenium cannot extract elements that are not displayed? Or am I doing something wrong here? Please advise.

Just in case you need the full code, this is the page: Singapore Streetdirectory

Cadell Teng
  • 224
  • 3
  • 23
  • For future questions, make sure you include all the necessary code to reproduce your issue - https://stackoverflow.com/help/minimal-reproducible-example. See the code in my answer as an example. – Andrew Guy Jan 31 '20 at 05:03

1 Answers1

1

The visibility_of_element_located method is for "checking that an element is present on the DOM of a page and visible.".

Clearly this element is not visible, as the element has style="display:none" set.

You could use presence_of_element_located instead, which will just check that the element is present in the DOM tree:

wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'current_address_name')))

Fixing a few other issues with your code, the full minimal, reproducible example would be:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium import webdriver

all_street_address = []

driver = webdriver.Chrome()
wait = WebDriverWait

driver.get("https://www.streetdirectory.com/sg/4-third-avenue-266576/1_27869.html")

wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'current_address_name')))
full_address = driver.find_element_by_id('current_address_name')
street_address = full_address.get_attribute('innerHTML').split('(S)')[0]
print('Street Address: ' + str(street_address))
all_street_address.append(street_address)
Andrew Guy
  • 9,310
  • 3
  • 28
  • 40
  • Thanks. Now it's able to find the element but unable to extract any text. Below is my terminal output. `Your search parameter is 068808` `Street Address: ` – Cadell Teng Jan 31 '20 at 04:05
  • Updated answer to include a fix for this. – Andrew Guy Jan 31 '20 at 05:01
  • Amazing. Works like magic. So, for non-visible element scraping, I need to use this method? Any recommendation to read up on `.get_attribute`? Thanks so much! – Cadell Teng Jan 31 '20 at 05:09
  • This answer might give a bit more detail: https://stackoverflow.com/a/40416415/4497519 – Andrew Guy Jan 31 '20 at 06:36