Python Selenium access inner elements

Question

Just for fun, I try to extract the scheduled games for day 1 of this NFL season. As of now, I have this code in Python

from selenium import webdriver

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

from webdriver_manager.chrome import ChromeDriverManager

browser = webdriver.Chrome(ChromeDriverManager().install())

from selenium.webdriver.chrome.options import Options

opts = Options()
opts.headless=True

#browser = driver(options=opts)
print("Start")

website = "https://www.nfl.com/schedules/2020/REG1"
browser.get(website)
gamedays = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "nfl-o-matchup-group")))
   
print ("There are: ", len(gamedays))

This prints "There are: 3" which is correct because there are three sections having that class. How can I dig deeper into each section? For example, there is an element having the game date and of course, there are elements having the home team and the away team.

I have tried

   for j in range(0,len(gamedays)):
      game_on = gamedays[j].find_elements_by_class_name('d3-o-section-title')
      print(game_on)

which prints

[<selenium.webdriver.remote.webelement.WebElement (session="d8807c1ca013d7a2d58bd7377b42ca1a", element="034de32d-bf64-4544-94aa-d97ed6640367")>]

That is not helpful.

So, how can I get to that information?

Any help is appreciated.

score 1 · Accepted Answer · answered Sep 08 '20 at 01:39

1

Solution

Loop through objects returned in game_on and print their innerHtml

print(game_on.get_attribute("innerHTML"))

OR

for element in game_on:
   print(element.get_attribute("outerHTML"))

answered Sep 08 '20 at 01:39

pygeek

7,356
1
20
41

The for loop did it. Thank you. – Michael Sep 12 '20 at 17:32

undetected Selenium · Answer 2 · 2020-09-12T20:34:53.377

1

You were almost there. Using the line of code:

print(game_on)

you were printing the element and you need to print the textContent instead.

Solution

To print the textContent e.g. Friday, September 11th, etc you can use either of the following Locator Strategies:

Using xpath and get_attribute():

browser.get(website)
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "nfl-o-matchup-group")))])

Using xpath and text attribute:

browser.get(website)
print([my_elem.text for my_elem in WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "nfl-o-matchup-group")))])

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Outro

Link to useful documentation:

get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

edited Sep 12 '20 at 20:34

answered Sep 08 '20 at 12:39

undetected Selenium

183,867
41
278
352

Hm, print(game_on.text) shows AttributeError: 'list' object has no attribute 'text' print(game_on.get_attribute("innerHTML")) shows AttributeError: 'list' object has no attribute 'get_attribute' Thanks for the links. – Michael Sep 12 '20 at 17:31
@Michael I assumed the `for` loop was already in your code trials, so I avoided it. Checkout the updated answer and let me know the status. – undetected Selenium Sep 12 '20 at 20:35
Thanks. I went with the other solution in the meantime. – Michael Sep 12 '20 at 21:14

Python Selenium access inner elements

2 Answers2

Solution

Solution

Outro