I ended up solving this using BrowserMob-Proxy. From their README:
BrowserMob Proxy allows you to manipulate HTTP requests and responses, capture HTTP content, and export performance data as a HAR file.
For Python3, I did the following:
Pre Requiements:
BrowserMob-Proxy installed and running
For Mac I used HomeBrew:
?> brew install browsermob-proxy
?> brew services start browsermob-proxy
Set up local python3 environment with pipenv (or your choice of virtual env manager)
?> brew install pipenv
?> pipenv --python 3.8
?> pipenv install browsermobproxy
?> pipenv install selenium
?> pipenv install ....
Ability to authenticate with data source for your webpage. Since I was utilizing a GCP service I followed the flow published in the IAP documentation for getting the authentication token found here: Authenticating from a service account
Simplified code for adding the proxy:
from selenium.webdriver import ChromeOptions
import browsermobproxy
# 1. Do whatever you need to do to get your token
token = get_auth_token()
# 2. Create browsermob client and add auth to headers
client = browsermobproxy.Client("localhost:9090") # port depends on your own setup
client.headers({"Authorization": "Bearer {}".format(token)})
# 3. Create browser (can vary wildly based on your own needs)
chrome_options = ChromeOptions()
chrome_options.add_argument("--ignore-certificate-errors") # I needed this, you may not
caps = chrome_options.to_capabilities()
client.add_to_capabilities(caps) # This is important!
# create driver instance with your capabilities