I am trying to use selenium-wire to replace headers when driving a headless chrome browser. I'm looking to run python code in a Docker container, thus the desire for a headless approach. I've written code based on this example: https://newbedev.com/setting-request-headers-in-selenium.
When I run the code below so as to observe the sent request, it is evident that the modifications to the headers is not being applied.
- User-Agent header still reports "HeadlessChrome/91.0.4472.114" instead of "Chrome/93.0.4577.63"
- The 3 new 'sec-ch' headers are not added to the request.
Any ideas why this might be happening and how to fix this?
My code is as follows:
from time import sleep
import requests
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
# Create a request interceptor
def interceptor(request):
del request.headers['User-Agent'] # Delete the header first
request.headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
request.headers['sec-ch-ua'] = '"Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"'
request.headers['sec-ch-ua-mobile'] = '?0'
request.headers['sec-ch-ua-platform'] = '"Linux"'
#sleep(0.1)
print("Starting a new web session...")
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument('--remote-debugging-address=0.0.0.0')
#driver = webdriver.Chrome("/usr/bin/chromedriver", options=chrome_options)
driver = webdriver.Chrome(options=chrome_options)
# Set the interceptor on the driver
driver.request_interceptor = interceptor
print("Getting website...")
driver.get('https://www.google.com/')
# Access requests via the `requests` attribute
for request in driver.requests:
print(request.headers)
print(request.response)
print("Waiting indefinitely...")
while True:
sleep(1)
The output on the docker is as follows:
worker | Starting a new web session...
worker | Getting website...
worker | {'Host': 'www.google.com', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.114 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'Sec-Fetch-Site': 'none', 'Sec-Fetch-Mode': 'navigate', 'Sec-Fetch-User': '?1', 'Sec-Fetch-Dest': 'document', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US'}
worker | 200 OK
worker | {'Host': 'www.google.com', 'Connection': 'keep-alive', 'sec-ch-ua': '', 'sec-ch-ua-mobile': '?0', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/91.0.4472.114 Safari/537.36', 'Accept': 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-Mode': 'no-cors', 'Sec-Fetch-Dest': 'image', 'Referer': 'https://www.google.com/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US', 'Cookie': 'CONSENT=PENDING+999'}
worker | 200 OK
... truncated to remove similar requests / responses
worker | Waiting indefinitely...
