How to scrape weight of player which is a hidden content using Selenium

Question

I have been trying to get the weight of this player, but it is only visible on the inspect element and not on the website, when I print the below, I get a blank result, Could anyone please help me with this?

Code trials:

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

s = Service('/Users/karim/Desktop/chromedriver-2')
driver = webdriver.Chrome(service=s)
url = 'https://www.premierleague.com/players/71432/Rayan-A%C3%AFt-Nouri/overview'
driver.get(url)

g = driver.find_element(By.XPATH, "//li[@class='u-hide']")
print(g.text)

score 1 · Answer 1 · answered Apr 29 '22 at 06:06

You need to do the following:

Click on accept cookies button.
Click on the close button that is on the top right-hand side.

Code:

driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://www.premierleague.com/players/71432/Rayan-A%C3%AFt-Nouri/overview")

try:
    wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept All Cookies']"))).click()
    print("clicked on accept cookies button")
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#advertClose"))).click()
except:
    pass

#print(wait.until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']"))).get_attribute('innerText'))
print(wait.until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerText'))

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

clicked on accept cookies button
70kg

Process finished with exit code 0

What was the reason, can you explain a bit more ? I applied the same cookies action but that did not solve it. How did you come up with the advert solution, it's not even showing up on the page. Any clarification is appreciated. — BcK, Apr 29 '22 at 09:24
When you try to get the weight from the web page manually, you will have to do this `1. Click on accept cookies button. 2. Click on the close button that is on the top right-hand side.` and then use the `get_attribute('innerText')` on the web element. — cruisepandey, Apr 29 '22 at 09:28
I didn't need to do anything of sort in order to get the weight manually from the page. I don't see any ad either, tried closing adblockers and all but to no avail. Strange. — BcK, Apr 29 '22 at 10:55

undetected Selenium · Answer 2 · 2022-04-30T00:05:07.570

The weight of the player is within:

<div class="info">70kg</div>

which is within it's ancestor:

<li class="u-hide">
          <div class="label">Weight</div>
          <div class="info">70kg</div>
    </li>

where the <li> is having class as u-hide

As the element is a hidden element using text attribute won't work here and you have extract either the innerHTML or the innerText.

Solution

To print the value of the Weight you can use either of the following locator strategies:

Using innerHTML:

print(driver.find_element(By.XPATH, "//li[@class='u-hide']//div[@class='info']").get_attribute('innerHTML'))

Using innerText:

print(driver.find_element(By.XPATH, "//li[@class='u-hide']//div[@class='info']").get_attribute('innerText'))

Inducing WebDriverWait:

Using innerHTML:

print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerHTML'))

Using innerText:

print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerText'))

Console Output:
```
70kg
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to scrape weight of player which is a hidden content using Selenium

2 Answers2

Solution