Scraping tables from website using Selenium and Python

Question

I'm new to python and was trying to scrape data from a website. So far, i can successfully scrape the data however the output is given in a linear format instead of the table its pulled from. Could someone give me some pointers on how to get the correct output?

import datetime
import pandas as pd
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

chrome_options = Options()
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 2})
chrome_options.add_argument("-headless")
chrome = webdriver.Chrome('chromedriver', options = chrome_options)

today = datetime.date.today()
first = today.replace(day=1)
lastMonth = first - datetime.timedelta(days=1)

date_range = pd.date_range(start='2019-01-01', end=lastMonth, freq='MS')
date_range.strftime("%B/%Y")

date_list = [i.strftime('%B/%Y') for i in date_range]

base_url = "https://www.horseracing.net/tipsters/sky-sports-racing/alex-hammond/"

length = len(date_list)
i = 41 # set to 41 for testing purposes, I didnt want to loop all the tables until i sorted the output. 


while i < length:
    page_url = str(base_url)+""+str(date_list[i])
    chrome.get(page_url)
    EL2 = chrome.find_element(By.CLASS_NAME, 'main-table-container')
    print(EL2.text)
    i += 1

Current Output

Date
Pos
Tip
Track
Stake
Type
Odds
Result
30th
5th
Clipsham Gold
Yarmouth
1.0 WIN
Nap
9/4
LOST
29th
2nd
Cavendish
Bath
1.0 WIN
Nap
9/4
LOST
28th
2nd
King Of War
Brighton
1.0 WIN
Nap
2/1
LOST
27th
1st
The Vollan
Southwell
1.0 WIN
Nap
11/8
WON

Desired Output (all the sites appended into one dataframe, so I can export to an .xlsx)

screenshot

I recommend that looking for youtube tutorials https://www.youtube.com/watch?v=TTiXxN3bD-Y&list=UUsFz0IGS9qFcwrh7a91juPg&index=32 — Dejene T., Jul 09 '22 at 14:41

score 0 · Accepted Answer · answered Jul 09 '22 at 21:24

To create a DataFrame using Pandas from the This Month's Tips table within the website you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://www.horseracing.net/tipsters/sky-sports-racing/alex-hammond/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.cookie-accept-button.cookie-button-link"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.close-modal"))).click()
date = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-left-section']//span[@class='tiping-row-text' and contains(., '/')]")))]
pose = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-left-section']//div[@class='tiping-row-cell position-cell']//span")))]
tip = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-left-section']//div[@class='tiping-row-cell position-cell']//following::div[1]/span")))]
track = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-right-section']//span[text()='Track']//following::span[1]")))]
stake = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-right-section']//span[text()='Stake']//following::span[1]")))]
types = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-right-section']//span[text()='Type']//following::span[1]")))]
odd = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-right-section']//span[text()='Odds']//following::span[1]")))]
result = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='main-table-tiping-row data-type-nap']//section[@class='cell-row-right-section']//div[@class='tiping-row-cell mobile-cell-hide']/span")))]
df = pd.DataFrame(data=list(zip(date, pose, tip, track, stake, types, odd, result)), columns=['Date', 'Pos', 'Tip', 'Track', 'Stake', 'Type', 'Odds', 'Result'])
print(df)

Note : You have to add the following imports :

import pandas as pd          
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Console Output:

       Date   Pos               Tip         Track    Stake Type    Odds Result
0  08/07/22   3rd       Documenting          York  1.0 WIN  Nap    12/1   LOST
1  07/07/22   4th         Zechariah     Newmarket  1.0 WIN  Nap     9/4   LOST
2  06/07/22  10th       Lord Cherry      Yarmouth  1.0 WIN  Nap     9/2   LOST
3  05/07/22   3rd    Landen Calling     Uttoxeter  1.0 WIN  Nap    13/8   LOST
4  04/07/22    PU  De Barley Basket     Worcester  1.0 WIN  Nap     4/1   LOST
5  03/07/22   2nd     Caius Marcius  Market Rasen  1.0 WIN  Nap     9/2   LOST
6  02/07/22   6th          Chairman       Sandown  1.0 WIN  Nap  100/30   LOST
7  01/07/22   3rd      Rum Cocktail     Doncaster  1.0 WIN  Nap    11/4   LOST

Will give this a try tomorrow morning, thank you for the response — juney881, Jul 10 '22 at 19:03

Scraping tables from website using Selenium and Python

1 Answers1