I am fairly new to web scraping and decided to dive straight into the deep end. I want select any product and "all months" in a dropdown above the table from https://www.cmegroup.com/tools-information/quikstrike/options-calendar.html and extract the table data into a scv file. The problem araises because the website is dynamic (not all HTML code is displayed when clicking inspect sourse in browser) and generates the table in css (from what i managed to understand). I tried using Selenium to load the webpage, but I am getting an error.
[12508:8412:0216/220631.827:ERROR:ssl_client_socket_impl.cc(985)] handshake failed; returned -1, SSL error code 1, net_error -101
I am assuming this has to do with the webdriver initialisation and I need to give it some settings, just not sure which ones.
Here is the code:
from selenium import webdriver
from bs4 import BeautifulSoup
# Set up the Selenium driver
driver = webdriver.Chrome()
# Open the webpage
url = 'https://www.cmegroup.com/tools-information/quikstrike/options-calendar.html'
driver.get(url)
# Render the page and extract the HTML code
html = driver.page_source
# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
# Extract the data you want from the soup object
tables = soup.findAll("table")
print(tables)
# Close the Selenium driver
driver.quit()
I have tried going the short route and reproducing the requests made by the browser and catching the response with the HTML code (yes the request only returns HTML, not JSON), but this backfired as I couldnt reproduce payload. How do I get the data from the calendar?
