I am having trouble scraping a table
from selenium import webdriver
driver = webdriver.Chrome('C:\Drivers\chromedriver.exe')
driver.get("https://fantasyteamadvice.com/dfs/nba/ownership")
html = driver.page_source
tm.sleep(8)
tables = pd.read_html(html)
data = tables[0:]
driver.close()
data.to_csv("inputs/dk_ownership.csv")
data
Getting Error below ValueError: No tables found
Use webdriverwait and wait for the element visible and find the table element and get the outHTML from the table and then use pandas.
waitWD = WebDriverWait (driver, 10)
link = "https://fantasyteamadvice.com/dfs/nba/ownership"
driver.get (link)
html=waitWD.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"table[data-testid='ownershipTablenba']"))).get_attribute("outerHTML")
table = pd.read_html(html)[0]
print(table)
table.to_csv("inputs/dk_ownership.csv")
You need to import below libraries.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
import pandas as pd
output of the DF:
Unnamed: 0 Player Draftkings Price DK NBA Ownership Projection Fanduel Price FD NBA Ownership Projection
0 NaN Klay Thompson $8000 3.3% $8000 4.7%
1 NaN Harrison Barnes $4900 1.8% $4600 8.6%
2 NaN Draymond Green $6800 1.7% $6800 7.3%
3 NaN Dario Saric $3500 0.2% $4100 0.2%
4 NaN Trey Lyles $3200 1.1% $4200 0.9%
5 NaN Kevon Looney $4700 7.9% $5700 4.3%
6 NaN Cedi Osman $3100 0.2% $4500 0.2%
7 NaN Domantas Sabonis $9900 10.6% $9600 17.0%
8 NaN Caris Levert $4500 1.3% $5300 0.2%
9 NaN De'Aaron Fox $9000 3.9% $9400 7.4%
10 NaN Malik Monk $4800 3.4% $5500 2.7%
11 NaN Donovan Mitchell $9700 7.0% $9500 8.0%
12 NaN Jarrett Allen $6900 15.0% $7600 7.4%
13 NaN Donte Divincenzo $5500 6.0% $5300 13.0%
14 NaN Shai Gilgeous-Alexander $10200 15.6% $10700 27.2%
15 NaN Kevin Huerter $5300 4.4% $6000 3.3%
16 NaN Kenrich Williams $4100 3.0% $4700 2.7%
17 NaN Terence Davis $4400 0.2% $4800 0.2%
18 NaN Darius Garland $7700 13.0% $8400 17.0%
19 NaN Luguentz Dort $4500 4.2% $5400 0.7%
20 NaN Ty Jerome $3600 0.4% $4400 0.4%
21 NaN Jordan Poole $8300 6.8% $8300 10.3%
22 NaN Isaac Okoro $3400 1.5% $4200 1.0%
23 NaN Isaiah Joe $3800 3.6% $4400 1.8%
24 NaN Jonathan Kuminga $4200 10.0% $4100 16.8%
25 NaN Anthony Lamb $3000 3.3% $3600 1.0%
26 NaN Lindy Waters III $3100 0.1% $-- 0.0%
27 NaN Jeremiah Robinson-Earl $3300 1.0% $3900 0.3%
28 NaN Moses Moody $3000 1.0% $3600 0.4%
29 NaN Davion Mitchell $3000 0.9% $3700 0.2%
30 NaN Josh Giddey $7500 6.0% $7500 15.0%
31 NaN Evan Mobley $7300 4.5% $7400 13.5%
32 NaN Keegan Murray $4900 2.2% $5200 0.7%
33 NaN Jalen Williams $6100 0.9% $6000 5.2%
34 NaN Jaylin Williams $3400 0.4% $4700 0.1%
You were close enough. Instead of passing driver.page_source as the data to the read_html() you need to pass only the <table> element as follows:
Code block:
driver.get('https://fantasyteamadvice.com/dfs/nba/ownership')
data = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table[data-testid='ownershipTablenba']"))).get_attribute("outerHTML")
df = pd.read_html(data)
print(df)
driver.quit()
Console output:
[ Unnamed: 0 Player ... Fanduel Price FD NBA Ownership Projection
0 NaN Klay Thompson ... $8000 4.7%
1 NaN Harrison Barnes ... $4600 8.6%
2 NaN Draymond Green ... $6800 7.3%
3 NaN Dario Saric ... $4100 0.2%
4 NaN Trey Lyles ... $4200 0.9%
5 NaN Kevon Looney ... $5700 4.3%
6 NaN Cedi Osman ... $4500 0.2%
7 NaN Domantas Sabonis ... $9600 17.0%
8 NaN Caris Levert ... $5300 0.2%
9 NaN De'Aaron Fox ... $9400 7.4%
10 NaN Malik Monk ... $5500 2.7%
11 NaN Donovan Mitchell ... $9500 8.0%
12 NaN Jarrett Allen ... $7600 7.4%
13 NaN Donte Divincenzo ... $5300 13.0%
14 NaN Shai Gilgeous-Alexander ... $10700 27.2%
15 NaN Kevin Huerter ... $6000 3.3%
16 NaN Kenrich Williams ... $4700 2.7%
17 NaN Terence Davis ... $4800 0.2%
18 NaN Darius Garland ... $8400 17.0%
19 NaN Luguentz Dort ... $5400 0.7%
20 NaN Ty Jerome ... $4400 0.4%
21 NaN Jordan Poole ... $8300 10.3%
22 NaN Isaac Okoro ... $4200 1.0%
23 NaN Isaiah Joe ... $4400 1.8%
24 NaN Jonathan Kuminga ... $4100 16.8%
25 NaN Anthony Lamb ... $3600 1.0%
26 NaN Lindy Waters III ... $-- 0.0%
27 NaN Jeremiah Robinson-Earl ... $3900 0.3%
28 NaN Moses Moody ... $3600 0.4%
29 NaN Davion Mitchell ... $3700 0.2%
30 NaN Josh Giddey ... $7500 15.0%
31 NaN Evan Mobley ... $7400 13.5%
32 NaN Keegan Murray ... $5200 0.7%
33 NaN Jalen Williams ... $6000 5.2%
34 NaN Jaylin Williams ... $4700 0.1%
[35 rows x 6 columns]]
Note : You have to add the following imports :
import pandas as pd
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC