Not getting values from cells after scraping a table from a dynamic webpage using Selenium

Question

I have the following code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('C:\chromedriver_win32\chromedriver.exe')
global_dynamicUrl = "https://draft.shgn.com/nfc/public/dp/788/grid"
driver.get(global_dynamicUrl)
wait = WebDriverWait(driver, 15)
table_entries = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table tr")))
print(len(table_entries))
print(table_entries)
html = driver.page_source
open('site.txt', 'wt').write(html)

with open('site.txt', encoding='utf-8') as f:
    lines = f.read()
driver.close()

The resulting scrape is saved in a text file for further analysis. The problem is that when checking or parsing what was scraped, there is no trace of the names that appear in the rendered webpage (you can check the pic). What could I be doing wrong?

What is the intent of your final code? Right now it looks like you are trying to grab the entire source of the page... and then what? — JeffC, Feb 28 '23 at 22:07
I want to get the names that appear in each "cell" of the table. The idea is to then create a dataframe with the "Team X" headers and the names in each column as they appear. — Carlos Marcano, Feb 28 '23 at 22:36

undetected Selenium · Accepted Answer · 2023-02-28T23:01:53.973

A minor adjustment would make your program perfecto. Instead of (By.CSS_SELECTOR, "table tr") induce WebDriverWait for visibility_of_all_elements_located() for the (By.CSS_SELECTOR, "table tr span.lname.ng-binding.medium") elements as follows:

Code block:

driver.get('https://draft.shgn.com/nfc/public/dp/788/grid')
table_entries = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table tr span.lname.ng-binding.medium")))
print(len(table_entries))
print(table_entries)
html = driver.page_source
open('site.txt', 'wt').write(html)
driver.quit()

Text file contents:

<span class="lname ng-binding medium" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Acuna Jr.</span>
.
<span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Trout</span>
.
<span class="team ng-binding">NYM<!-- ngIf: ::(dbs.sport === 'NFL') --></span></span> <span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Marte</span>
.
<span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Doval</span>

Thanks, I tried the change but no luck, the names inside the cells don't make it to the scraped info. — Carlos Marcano, Feb 28 '23 at 22:37
@CarlosMarcano Ofcoarse it does. I did check with the first and the last names respectively **Acuna Jr.** and **Doval**. Checkout the updated answer. — undetected Selenium, Feb 28 '23 at 22:40
Thanks. This is interesting: if you look for those you showed, they appear easily but if I look for "Trout" or "Marte" (which appear on the browser), I can't get them in the text file. — Carlos Marcano, Feb 28 '23 at 22:51
@CarlosMarcano Even "Trout" or "Marte" is there. Did you run the code block in this answer as it is? — undetected Selenium, Feb 28 '23 at 23:02

Not getting values from cells after scraping a table from a dynamic webpage using Selenium

1 Answers1