How to scrape in Python Selenium?

Question

If you visit this site, https://www.premierleague.com/match/66686 and then press stats tab, you will see several information about the match. How am I supposed to scrape the Possession for both teams?

This did not work.

stats = driver.find_element(By.XPATH, '//*[@id="mainContent"]/div/section[2]/div[2]/div[2]/div[1]/div/div/ul/li[3]')
stats.click()
    posHome = driver.find_element(By.XPATH,'//body[1]/main[1]/div[1]/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table[1]/tbody[1]/tr[1]/td[1]')
    print(posHome.text)
    posAway = driver.find_element(By.XPATH,'//*[@id="mainContent"]/div/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table/tbody/tr[1]/td[3]')
    print(posAway.text)

Please let me know how to solve this issue and thanks!

score 0 · Answer 1 · answered May 01 '22 at 22:17

To print the Possession for both teams you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies:

Code Block:

driver.get("https://www.premierleague.com/match/66686")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept All Cookies']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[text()='Stats']"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td>p"))).text)
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td:nth-child(3)>p"))).text)
driver.quit()

Console Output:
```
33.9
66.1
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

How to scrape in Python Selenium?

1 Answers1