How to scrape the list of image urls from college football data using Selenium by class_name

Question

I'm trying to scrape player data from college football roster sites. I am primarily interested in getting the player image, weight, and name. I have already been able to extract the weight and name but am struggling on extracting the image using selenium. This is my code so far.

driver = webdriver.Chrome("C:/Users/<my_user>/Downloads/chromedriver.exe")
driver.get(school["url"])
all_names = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-name')
all_weights = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-weight')
all_imgs = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-image')

This is an example of what would be passed in as school[url]. Many colleges use this format. https://rolltide.com/sports/football/roster Each player on this site has the following html element.

<div class="sidearm-roster-player-image column">

  <a data-bind="click: function() { return true; }, clickBubble: false" href="/sports/football/roster/jeremiah-alexander/8141" aria-label="Jeremiah Alexander - View Full Bio" title="View Full Bio">
    <img class=" lazyloaded" data-src="https://d1a8hwz3c6qyrc.cloudfront.net/images/2022/3/1/Alexander_Jeremiah.jpg?width=80" alt="Jeremiah Alexander" src="https://d1a8hwz3c6qyrc.cloudfront.net/images/2022/3/1/Alexander_Jeremiah.jpg?width=80">
  </a>

</div>

The issue I am running into is that each webElement in all_imgs does not seem to have an attribute such called 'img' or any attribute I can see that represents the image link located within the element. How can I get the link of all the images of the players on this page?

undetected Selenium · Answer 1 · 2022-03-24T20:18:09.357

To create a list with all the values of the src attribute you can use list comprehension and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.get('https://rolltide.com/sports/football/roster')
print([my_elem.get_attribute("src") for my_elem in driver.find_elements(By.CSS_SELECTOR, "a[title='View Full Bio'] > img")])

Using XPATH:

driver.get('https://rolltide.com/sports/football/roster')
print([my_elem.get_attribute("src") for my_elem in driver.find_elements(By.XPATH, "//a[@title='View Full Bio']/img")])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

score 2 · Accepted Answer · answered Mar 24 '22 at 20:17

Actually... you can get all the data without selenium. Everything is in the source HTML.

Here's how to do this:

import json

import requests
from bs4 import BeautifulSoup
from tabulate import tabulate

url = "https://rolltide.com/sports/football/roster"

soup = BeautifulSoup(requests.get(url).text, "lxml")
height_weight = [
    " ".join(i.getText(strip=True, separator=" ").split()) for i in
    soup.select(".sidearm-roster-default-template .sidearm-roster-player-position")
]

images_and_names = [
    [
        player['name'],
        player['url'],
        player['image']['url']
    ] for player in
    json.loads(soup.find("script", {"type": "application/ld+json"}).string)["item"]
]

player_data = [[i, *j] for i, j in zip(height_weight, images_and_names)]
print(tabulate(player_data, headers=["Body Info", "Name", "Full Bio", "Image"]))

This should output:

Body Info           Name                  Full Bio                                    Image
------------------  --------------------  ------------------------------------------  -------------------------------------------------------------
LB 6'2" 258 lbs     Jeremiah Alexander    http://rolltide.com/roster.aspx?rp_id=8141  http://rolltide.com/images/2022/3/1/Alexander_Jeremiah.jpg
PK 6'2" 188 lbs     Chase Allen           http://rolltide.com/roster.aspx?rp_id=8048  http://rolltide.com/images/2020/7/8/Allen_Chase.jpg
WR 5'9" 184 lbs     Aaron Anderson        http://rolltide.com/roster.aspx?rp_id=8142  http://rolltide.com/images/2022/3/1/Anderson_Aaron.jpg
LB 6'4" 243 lbs     Will Anderson Jr.     http://rolltide.com/roster.aspx?rp_id=8049  http://rolltide.com/images/2020/3/19/Anderson_Jr_Will_71.jpg
DB 6'0" 188 lbs     Terrion Arnold        http://rolltide.com/roster.aspx?rp_id=8050  http://rolltide.com/images/2021/8/6/Aronld_Terrion.jpg
DL 6'5" 305 lbs     Anquin Barnes Jr.     http://rolltide.com/roster.aspx?rp_id=8051  http://rolltide.com/images/2021/8/6/Barnes_Anquin.jpg
DB 6'1" 206 lbs     Jordan Battle         http://rolltide.com/roster.aspx?rp_id=8052  http://rolltide.com/images/2020/3/19/Battle_Jordan_71.jpg
RB 5'8" 180 lbs     Jonathan Bennett      http://rolltide.com/roster.aspx?rp_id=8053  http://rolltide.com/images/2020/11/2/Bennett_Jonathan_71.jpg
LB 6'2" 233 lbs     Kendrick Blackshire   http://rolltide.com/roster.aspx?rp_id=8054  http://rolltide.com/images/2021/8/6/Blackshire_Kendrick.jpg
WR 6'0" 176 lbs     Bret Bolin            http://rolltide.com/roster.aspx?rp_id=8055  http://rolltide.com/images/2020/7/8/Bolin_Bret.jpg
OL 6'5" 332 lbs     Tyler Booker          http://rolltide.com/roster.aspx?rp_id=8143  http://rolltide.com/images/2022/3/1/Booker_Tyler.jpg
OL 6'5" 293 lbs     Tanner Bowles         http://rolltide.com/roster.aspx?rp_id=8056  http://rolltide.com/images/2020/3/19/Bowles_Tanner_71.jpg
WR 5'11" 182 lbs    Jacoby Boykins        http://rolltide.com/roster.aspx?rp_id=8057  http://rolltide.com/images/2021/8/6/Boykins_Jacoby.jpg
DB 6'0" 193 lbs     Brian Branch          http://rolltide.com/roster.aspx?rp_id=8058  http://rolltide.com/images/2021/6/17/Branch_Brian.jpg
LB 6'3" 240 lbs     Chris Braswell        http://rolltide.com/roster.aspx?rp_id=8059  http://rolltide.com/images/2020/3/19/Braswell_Chris_71.jpg
OL 6'3" 282 lbs     James Brockermeyer    http://rolltide.com/roster.aspx?rp_id=8060  http://rolltide.com/images/2021/2/9/Brockermeyer_James_71.jpg
OL 6'5" 304 lbs     Tommy Brockermeyer    http://rolltide.com/roster.aspx?rp_id=8061  http://rolltide.com/images/2021/2/9/Brockermeyer_Tommy.jpg
WR 6'2" 196 lbs     Ja'Corey Brooks       http://rolltide.com/roster.aspx?rp_id=8062  http://rolltide.com/images/2021/2/9/Brooks_Ja_Corey.jpg
TE 6'5" 238 lbs     Elijah Brown          http://rolltide.com/roster.aspx?rp_id=8144  http://rolltide.com/images/2022/3/1/Brown_Elijah.jpg
WR 6'3" 175 lbs     J'Kolbe Bulock        http://rolltide.com/roster.aspx?rp_id=8145  http://rolltide.com/images/2022/3/1/Bulock_J_Kolbe.jpg
P 6'6" 211 lbs      James Burnip          http://rolltide.com/roster.aspx?rp_id=8063  http://rolltide.com/images/2021/8/6/Burnip_James.jpg
DL 6'3" 309 lbs     Jamil Burroughs       http://rolltide.com/roster.aspx?rp_id=8064  http://rolltide.com/images/2020/7/8/Burroughs_Jamil.jpg
WR 6'0" 200 lbs     Jermaine Burton       http://rolltide.com/roster.aspx?rp_id=8146  http://rolltide.com/images/2022/3/1/Burton_Jermaine.jpg
LB 6'3" 225 lbs     Jihaad Campbell       http://rolltide.com/roster.aspx?rp_id=8147  http://rolltide.com/images/2022/3/1/Campbell_Jihaad.jpg
OL 6'4" 305 lbs     Javion Cohen          http://rolltide.com/roster.aspx?rp_id=8065  http://rolltide.com/images/2020/7/8/Cohen_Javion.jpg
RB 5'11" 210 lbs    Elijah Crockett       http://rolltide.com/roster.aspx?rp_id=8067  http://rolltide.com/images/2021/8/20/Crockett_Elijah.jpg
OL 6'3" 305 lbs     Darrian Dalcourt      http://rolltide.com/roster.aspx?rp_id=8068  http://rolltide.com/images/2020/3/19/Dalcourt_Darrian_53.jpg
DL 6'3" 300 lbs     DJ Dale               http://rolltide.com/roster.aspx?rp_id=8069  http://rolltide.com/images/2021/8/20/Dale_DJ.jpg
WR 5'10" 177 lbs    JoJo Earle            http://rolltide.com/roster.aspx?rp_id=8070  http://rolltide.com/images/2021/8/6/Earle_JoJo.jpg
DL 6'5" 292 lbs     Justin Eboigbe        http://rolltide.com/roster.aspx?rp_id=8071  http://rolltide.com/images/2020/8/17/Eboigbe_Justin.jpg
OL 6'3" 307 lbs     Emil Ekiyor Jr.       http://rolltide.com/roster.aspx?rp_id=8073  http://rolltide.com/images/2020/8/17/Ekiyor_Jr_Emil.jpg
TE 6'0" 220 lbs     Robert Ellis          http://rolltide.com/roster.aspx?rp_id=8074  http://rolltide.com/images/2020/11/2/Ellis_Robert.jpg
DB 6'2" 185 lbs     Tre'Quon Fegans       http://rolltide.com/roster.aspx?rp_id=8148  http://rolltide.com/images/2022/3/1/Fegans_Tre_Quon.jpg
OL 6'4" 300 lbs     Terrence Ferguson II  http://rolltide.com/roster.aspx?rp_id=8075  http://rolltide.com/images/2021/2/9/Ferguson_Terrence.jpg
LB 6'0" 212 lbs     Kyle Flood Jr.        http://rolltide.com/roster.aspx?rp_id=8076  http://rolltide.com/images/2022/3/1/Flood_Jr_Kyle.jpg
OL 6'6" 333 lbs     Damieon George Jr.    http://rolltide.com/roster.aspx?rp_id=8077  http://rolltide.com/images/2020/7/8/George_Jr_Damieon.jpg
RB 5'11" 200 lbs    Jahmyr Gibbs          http://rolltide.com/roster.aspx?rp_id=8149  http://rolltide.com/images/2022/3/1/Gibbs_Jahmyr.jpg
DL 6'4" 288 lbs     Monkell Goodwine      http://rolltide.com/roster.aspx?rp_id=8078  http://rolltide.com/images/2021/2/9/Goodwine_Monkell.jpg
WR 6'3" 202 lbs     Agiye Hall            http://rolltide.com/roster.aspx?rp_id=8079  http://rolltide.com/images/2021/2/9/Hall_Agiye.jpg
DB 6'1" 208 lbs     DeMarcco Hellams      http://rolltide.com/roster.aspx?rp_id=8080  http://rolltide.com/images/2020/3/19/Hellams_DeMarcco_58.jpg
SN 6'2" 245 lbs     Kneeland Hibbett      http://rolltide.com/roster.aspx?rp_id=8081  http://rolltide.com/images/2021/8/6/Hibbett_Kneeland.jpg
WR 6'3" 214 lbs     Traeshon Holden       http://rolltide.com/roster.aspx?rp_id=8082  http://rolltide.com/images/2020/3/19/Holden_Traeshon_53.jpg
DL 6'4" 295 lbs     Braylen Ingraham      http://rolltide.com/roster.aspx?rp_id=8084  http://rolltide.com/images/2020/3/19/Ingraham_Braylen_53.jpg
LB 6'1" 235 lbs     Ian Jackson           http://rolltide.com/roster.aspx?rp_id=8085  http://rolltide.com/images/2021/2/9/Jackson_Ian.jpg
DB 6'3" 198 lbs     Khyree Jackson        http://rolltide.com/roster.aspx?rp_id=8086  http://rolltide.com/images/2021/8/6/Jackson_Khyree.jpg
QB 6'2" 186 lbs     Blake Jarrett         http://rolltide.com/roster.aspx?rp_id=8150  http://rolltide.com/images/2022/3/1/Jarrett_Blake.jpg
LB 6'5" 230 lbs     Christian Johnson     http://rolltide.com/roster.aspx?rp_id=8087  http://rolltide.com/images/2021/8/16/Johnson_Christian.jpg
WR 6'0" 190 lbs     Thaiu Jones-Bell      http://rolltide.com/roster.aspx?rp_id=8089  http://rolltide.com/images/2020/3/19/Jones_Bell_Thaiu_71.jpg
DL 6'2" 343 lbs     Tim Keenan III        http://rolltide.com/roster.aspx?rp_id=8090  http://rolltide.com/images/2021/8/6/Keenan_III_Tim.jpg
LB 6'3" 220 lbs     Demouy Kennedy        http://rolltide.com/roster.aspx?rp_id=8091  http://rolltide.com/images/2020/8/20/Kennedy_Demouy.jpg
OL 6'7" 322 lbs     Amari Kight           http://rolltide.com/roster.aspx?rp_id=8092  http://rolltide.com/images/2020/3/19/Kight_Amari_81.jpg
LB 6'4" 231 lbs     Keanu Koht            http://rolltide.com/roster.aspx?rp_id=8093  http://rolltide.com/images/2021/2/9/Koht_Keanu.jpg
WR 6'2" 192 lbs     Grant Krieger         http://rolltide.com/roster.aspx?rp_id=8094  http://rolltide.com/images/2020/3/19/Krieger_Grant_30.jpg
DB 6'1" 170 lbs     Brylan Lanier         http://rolltide.com/roster.aspx?rp_id=8095  http://rolltide.com/images/2021/8/20/Lanier_Brylan.jpg
DL 6'3" 278 lbs     Jah-Marien Latham     http://rolltide.com/roster.aspx?rp_id=8096  http://rolltide.com/images/2020/7/8/Latham_Jah_Marien.jpg
OL 6'6" 326 lbs     JC Latham             http://rolltide.com/roster.aspx?rp_id=8097  http://rolltide.com/images/2021/2/9/Latham_JC.jpg
TE 6'5" 244 lbs     Cameron Latu          http://rolltide.com/roster.aspx?rp_id=8098  http://rolltide.com/images/2020/3/19/Latu_Cameron_53.jpg
WR 5'11" 193 lbs    Kendrick Law          http://rolltide.com/roster.aspx?rp_id=8151  http://rolltide.com/images/2022/3/1/Law_Kendrick.jpg
LB 6'2" 225 lbs     Deontae Lawson        http://rolltide.com/roster.aspx?rp_id=8099  http://rolltide.com/images/2021/2/9/Lawson_Deontae.jpg
WR 5'10" 175 lbs    Christian Leary       http://rolltide.com/roster.aspx?rp_id=8100  http://rolltide.com/images/2021/2/9/Leary_Christian.jpg
P 6'2" 207 lbs      Jack Martin           http://rolltide.com/roster.aspx?rp_id=8102  http://rolltide.com/images/2021/8/6/Martin_Jack.jpg
DB 6'1" 143 lbs     Jacobi McBride        http://rolltide.com/roster.aspx?rp_id=8179  http://rolltide.com/images/2021/3/3/McBride_Jacobi.jpg
RB 5'11" 212 lbs    Jase McClellan        http://rolltide.com/roster.aspx?rp_id=8103  http://rolltide.com/images/2020/3/19/McClellan_Jase_58.jpg
DB 6'1" 188 lbs     Kool-Aid McKinstry    http://rolltide.com/roster.aspx?rp_id=8104  http://rolltide.com/images/2021/2/9/McKinstry_Ga_Quincy.jpg
OL 6'4" 295 lbs     Seth McLaughlin       http://rolltide.com/roster.aspx?rp_id=8105  http://rolltide.com/images/2020/5/1/McLaughlin_Seth.jpg
RB 5'10" 201 lbs    Jamarion Miller       http://rolltide.com/roster.aspx?rp_id=8152  http://rolltide.com/images/2022/3/1/Miller_Jamarion.jpg
QB 6'2" 212 lbs     Jalen Milroe          http://rolltide.com/roster.aspx?rp_id=8106  http://rolltide.com/images/2021/2/9/Milroe_Jalen.jpg
LB 6'2" 225 lbs     Jaylen Moody          http://rolltide.com/roster.aspx?rp_id=8107  http://rolltide.com/images/2020/3/19/Moody_Jaylen_53.jpg
DB 6'0" 190 lbs     Malachi Moore         http://rolltide.com/roster.aspx?rp_id=8108  http://rolltide.com/images/2020/7/8/Moore_Malachi.jpg
LB 6'2" 224 lbs     Shawn Murphy          http://rolltide.com/roster.aspx?rp_id=8153  http://rolltide.com/images/2022/3/1/Murphy_Shawn.jpg
DL 6'5" 370 lbs     Jaheim Oatis          http://rolltide.com/roster.aspx?rp_id=8154  http://rolltide.com/images/2022/3/1/Oatis_Jaheim.jpg
TE 6'4" 258 lbs     Robbie Ouzts          http://rolltide.com/roster.aspx?rp_id=8109  http://rolltide.com/images/2021/2/9/Ouzts_Robbie.jpg
DL 6'4" 303 lbs     Damon Payne Jr.       http://rolltide.com/roster.aspx?rp_id=8111  http://rolltide.com/images/2021/8/6/Payne_Damon.jpg
DL 6'2" 264 lbs     Khurtiss Perry        http://rolltide.com/roster.aspx?rp_id=8155  http://rolltide.com/images/2022/3/1/Perry_Khurtiss.jpg
TE 6'6" 232 lbs     Jax Porter            http://rolltide.com/roster.aspx?rp_id=8156  http://rolltide.com/images/2022/3/1/Porter_Jax_71.jpg
DB 6'0" 175 lbs     Blake Pugh            http://rolltide.com/roster.aspx?rp_id=8112  http://rolltide.com/images/2021/8/20/Pugh_Blake.jpg
SN 6'5" 273 lbs     Gabe Pugh             http://rolltide.com/roster.aspx?rp_id=8113  http://rolltide.com/images/2021/8/16/Pugh_Gabe.jpg
DL 6'1" 236 lbs     Chase Quigley         http://rolltide.com/roster.aspx?rp_id=8114  http://rolltide.com/images/2021/8/6/Quigley_Chase.jpg
TE/OL 6'4" 298 lbs  Kendall Randolph      http://rolltide.com/roster.aspx?rp_id=8115  http://rolltide.com/images/2020/3/19/Randolph_Kendall_29.jpg
PK 6'1" 190 lbs     Will Reichard         http://rolltide.com/roster.aspx?rp_id=8116  http://rolltide.com/images/2020/8/17/Reichard_Will.jpg
DB 6'2" 190 lbs     Eli Ricks             http://rolltide.com/roster.aspx?rp_id=8157  http://rolltide.com/images/2022/3/1/Ricks_Eli.jpg
OL 6'5" 302 lbs     Jaeden Roberts        http://rolltide.com/roster.aspx?rp_id=8117  http://rolltide.com/images/2021/8/6/Roberts_Jaeden.jpg
DB 6'2" 197 lbs     Jahquez Robinson      http://rolltide.com/roster.aspx?rp_id=8118  http://rolltide.com/images/2020/3/19/Robinson_Jahquez_30.jpg
LB 6'5" 224 lbs     Quandarrius Robinson  http://rolltide.com/roster.aspx?rp_id=8119  http://rolltide.com/images/2020/7/8/Robinson_Quandarrius.jpg
OL 6'5" 285 lbs     Jackson Roby          http://rolltide.com/roster.aspx?rp_id=8120  http://rolltide.com/images/2020/3/19/Roby_Jackson_71.jpg
OL 6'3" 285 lbs     Graham Roten          http://rolltide.com/roster.aspx?rp_id=8121  http://rolltide.com/images/2021/8/6/Roten_Graham.jpg
RB 6'0" 214 lbs     Trey Sanders          http://rolltide.com/roster.aspx?rp_id=8122  http://rolltide.com/images/2020/8/17/Sanders_Trey.jpg
PK 6'0" 185 lbs     Reid Schuback         http://rolltide.com/roster.aspx?rp_id=8123  http://rolltide.com/images/2021/8/6/Schuback_Reid.jpg
OL 6'5" 308 lbs     Dayne Shor            http://rolltide.com/roster.aspx?rp_id=8158  http://rolltide.com/images/2022/3/1/Shor_Dayne.jpg
QB 6'2" 198 lbs     Ty Simpson            http://rolltide.com/roster.aspx?rp_id=8159  http://rolltide.com/images/2022/3/1/Simpson_Ty.jpg
TE 6'1" 232 lbs     Charlie Skehan        http://rolltide.com/roster.aspx?rp_id=8125  http://rolltide.com/images/2020/12/4/Skehan_Charlie.jpg
DB 6'0" 185 lbs     DeVonta Smith         http://rolltide.com/roster.aspx?rp_id=8126  http://rolltide.com/images/2021/8/6/Smith_DeVonta.jpg
LB 5'10" 210 lbs    Jordan Smith          http://rolltide.com/roster.aspx?rp_id=8127  http://rolltide.com/images/2021/8/16/Smith_Jordan.jpg
DL 6'4" 304 lbs     Tim Smith             http://rolltide.com/roster.aspx?rp_id=8128  http://rolltide.com/images/2020/7/8/Smith_Tim.jpg
DB 6'1" 211 lbs     Kristian Story        http://rolltide.com/roster.aspx?rp_id=8129  http://rolltide.com/images/2020/7/8/Story_Kristian.jpg
TE 6'5" 232 lbs     Adam Thorsland        http://rolltide.com/roster.aspx?rp_id=8130  http://rolltide.com/images/2021/8/6/Thorsland_Adam.jpg
LB 6'2" 228 lbs     Henry To'oTo'o        http://rolltide.com/roster.aspx?rp_id=8131  http://rolltide.com/images/2021/8/6/To_oto_o_Henry.jpg
LB 6'4" 240 lbs     Dallas Turner         http://rolltide.com/roster.aspx?rp_id=8132  http://rolltide.com/images/2021/8/6/Turner_Dallas.jpg
SN 5'9" 185 lbs     Kade Wehby            http://rolltide.com/roster.aspx?rp_id=8133  http://rolltide.com/images/2022/3/1/Wehby_Kade.jpg
LB 6'1" 222 lbs     Bennett  Whisenhunt   http://rolltide.com/roster.aspx?rp_id=8134  http://rolltide.com/images/2020/12/4/Whisenhunt_Bennett.jpg
DB 6'2" 203 lbs     Kaine Williams        http://rolltide.com/roster.aspx?rp_id=8135  http://rolltide.com/images/2021/8/6/Williams_Kaine.jpg
RB 5'10" 212 lbs    Roydell Williams      http://rolltide.com/roster.aspx?rp_id=8136  http://rolltide.com/images/2020/3/19/Williams_Roydell_53.jpg
WR 5'10" 165 lbs    Sam Willoughby        http://rolltide.com/roster.aspx?rp_id=8137  http://rolltide.com/images/2021/8/16/Willoughby_Sam.jpg
DL 6'4" 307 lbs     Stephon Wynn Jr.      http://rolltide.com/roster.aspx?rp_id=8138  http://rolltide.com/images/2020/3/19/Wynn_Jr_Stephon_71.jpg
QB 6'0" 194 lbs     Bryce Young           http://rolltide.com/roster.aspx?rp_id=8139  http://rolltide.com/images/2020/3/19/Young_Bryce_71.jpg
DL 6'3" 292 lbs     Byron Young           http://rolltide.com/roster.aspx?rp_id=8140  http://rolltide.com/images/2020/3/19/Young_Byron_76.jpg

So I want to use this on the 2020 roster which is located at https://rolltide.com/sports/football/roster/2020 Would you know why this code doesn't work on a seemingly identical page? — Hfhdj Hdhnf, Mar 25 '22 at 07:32

score 1 · Answer 3 · answered Mar 25 '22 at 10:54

Accept baduker's solution as it's correct. I am just adding on to it to answer your follow up.

The reason the 2020 roster is throwing an error is because some of the players do not have an image url. There's a few ways you could attack that. I simply just took the json that baduker parses, and priror to creating the player_data from that, we'll fill in None for the players missing an image url:

import json

import requests
from bs4 import BeautifulSoup
from tabulate import tabulate

url = "https://rolltide.com/sports/football/roster/2020"

soup = BeautifulSoup(requests.get(url).text, "lxml")
height_weight = [
    " ".join(i.getText(strip=True, separator=" ").split()) for i in
    soup.select(".sidearm-roster-default-template .sidearm-roster-player-position")
]


jsonData = json.loads(soup.find("script", {"type": "application/ld+json"}).string)["item"]
for idx, each in enumerate(jsonData):
    if each['image'] == None:
        jsonData[idx]['image'] = {'url':None}
        

images_and_names = [
    [
        player['name'],
        player['url'],
        player['image']['url']
    ] for player in
    jsonData
]

player_data = [[i, *j] for i, j in zip(height_weight, images_and_names)]
print(tabulate(player_data, headers=["Body Info", "Name", "Full Bio", "Image"]))

This makes sense, thank you! I realized playing around with baduker's code that the issue was being thrown by a single row not having an image as you said. After ignoring this value I was able to proceed. — Hfhdj Hdhnf, Mar 25 '22 at 18:18

How to scrape the list of image urls from college football data using Selenium by class_name

3 Answers3