How can I scrape the position of the players in Selenium?

Question

If you visit this site, https://www.premierleague.com/players, you will be able to see a list of players with their positions and Nationality. I am only required to scrape the position but I am unable to

playerss = driver.find_elements(By.XPATH, '//*[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr')

for player in playerss:
    position = player.find_element(By.XPATH,'//[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr/td[2]')
    print(position)

This is what I have but it is not working. If anyone knows how to help, I will appreciate it.

Thanks!

Would you consider beautifulsoup instead? – jabbson Apr 29 '22 at 23:18 — jabbson, Apr 29 '22 at 23:18
I don't know beautifulsoup, that's the problem. – CloneXI Apr 29 '22 at 23:19 — CloneXI, Apr 29 '22 at 23:19

jabbson · Answer 1 · 2022-04-29T23:54:33.297

1

This is how I was able to get the list of positions:

...
driver.get('https://www.premierleague.com/players')
positions = driver.find_elements(By.CSS_SELECTOR, 
                                 'table tbody tr td:nth-child(2)')

for pos in positions:
    print(pos.text)

if you have to use xpath:

positions = driver.find_elements(By.XPATH, '//table/tbody/tr/td[2]')

edited Apr 29 '22 at 23:54

answered Apr 29 '22 at 23:45

jabbson

4,390
1
13
23

score 0 · Answer 2 · answered Apr 29 '22 at 23:49

The position of the players are within:

<td class="hide-s">Midfielder</td>

where the <td> is having class as hide-s

Solution

To print the positions you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following locator strategies:

Using text attribute:

print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])

Using innerHTML:

print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])

Using innerText:

print([my_elem.get_attribute("innerText") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])

Console Output:

['Midfielder', 'Defender', 'Defender', 'Forward', 'Forward', 'Forward', 'Defender', 'Goalkeeper', 'Forward', 'Defender', 'Defender', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Goalkeeper', 'Midfielder', 'Midfielder', 'Midfielder', 'Defender', 'Midfielder', 'Forward', 'Forward']

How can I scrape the position of the players in Selenium?

2 Answers2

Solution