-2

I recently started with coding, I use Python and Pycharm. I Installed and imported the needed "Add-ons" like Selenium.
For my first project I tried to get the "address" information from this website:

https://randomstreetview.com/#fullscreen

When I use the Chrome developer tools (F12) the information shows up after the div'address'. After lots of trial and error I managed to scrape the website in multiple ways but nothing ever showed up behind the div.
Due to that I looked it up and think it's a dynamic information, so I started trying to use Selenium and other code that I found online.
Nothing worked or I didn't manage to get it working.
The address changes everytime when you open the website but the place stays the same (afaik).
Can anyone help me with a working code or help me to get on the right way?
I could provide my different approaches but I won't think that would help.
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Bartelds
  • 5
  • 1
  • 1
    Post the code you've tried. Or the HTML you'd like to access. Watchout for common pitfalls like iframes. – DMart Jan 04 '21 at 01:49
  • Welcome to Stack Overflow! Please read [ask], especially the part about [mcve] (MCVE), and [How much research effort is expected?](https://meta.stackoverflow.com/questions/261592/how-much-research-effort-is-expected-of-stack-overflow-users) This will help you debug your own programs and solve problems for yourself. If you do this and are still stuck you can come back and post your MCVE, what you tried, and the execution result including any error messages so we can better help you. Also provide a link to the page and/or the relevant HTML. – JeffC Jan 04 '21 at 04:23

3 Answers3

1

If you want the element's address just get the element and print it's text.

driver.get("https://randomstreetview.com/")
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.presence_of_element_located((By.ID, "address")))
print(elem.text)

Element

<div id="address">Nordre Ringvej 97, 2600 Glostrup, Dänemark</div>

Outputs

Nordre Ringvej 97, 2600 Glostrup, Dänemark

Imports

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
Arundeep Chohan
  • 9,779
  • 5
  • 15
  • 32
1

To print the text value you can use either of the following Locator Strategies:

  • Using id and get_attribute("textContent"):

    driver.get("https://randomstreetview.com/#fullscreen")
print(driver.find_element_by_id("address").get_attribute("textContent"))

  • Using css_selector and get_attribute("innerHTML"):

    driver.get("https://randomstreetview.com/#fullscreen")
print(driver.find_element_by_css_selector("div#address").get_attribute("innerHTML"))

  • Using xpath and text attribute:

    driver.get("https://randomstreetview.com/#fullscreen")
print(driver.find_element_by_xpath("//div[@id='address']").text)

Ideally you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following Locator Strategies:

  • Using ID and get_attribute("textContent"):

    print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "address"))).get_attribute("textContent"))

  • Using CSS_SELECTOR and text attribute:

    print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#address"))).text)

  • Using XPATH and get_attribute():

    print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//div[@id='address']"))).get_attribute("innerHTML"))

  • Console Output:

    value

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

  • Console Output:

    Ciudad Pérdida 10, La Sabana, 39799 Acapulco, Guerrero, Mexico

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

References

Link to useful documentation:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
0

To piggy back on Arundeep Chohan's answer. The reason that you are unable to get the address is because it is a hidden element.

check out this post Python Selenium: Finds h1 element but returns empty text string

TLDR; "text property allow you to get text from only visible elements while textContent attribute also allow to get text of hidden one..."

This code also works using CSS selectors

element = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div#address')))

print(element.get_attribute('textContent'))
DBraun
  • 38
  • 1
  • 7