1

I have tried using the webbrowser module, with the code below but want to set custom headers such as:

'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1'

Here's the code I have right now:

import webbrowser
webbrowser.open("https://www.bing.com/search?q=TEST123")

I'm okay with using another library too. Essentially I want my python script to open a url in my default browser with custom headers.

Hammaad
  • 193
  • 1
  • 2
  • 14
  • @Sujay I would like my default browser to open that url with the custom header I am setting. – Hammaad May 23 '21 at 17:14
  • Do you need to specifically open it in the browser? Or are you okay with making a request (if you're scraping, or downloading, or whatever)? Passing headers with the `webbrowser` module isn't possible. – Safwan Samsudeen May 24 '21 at 09:17

1 Answers1

2

The documentation for the webbrowser module doesn't provide information on how to access underlying headers. It seems this is not possible. As stated in the documentation:

The webbrowser module provides a high-level interface to allow displaying Web-based documents to users.

  1. Use an extension/addon on your browser

You could use your current code and install an extension on your browser like simple-modify-headers extension for Firefox or Chrome. (The extension can be installed via this link for Firefox and via this link for Chrome).

It's very easy to change header values with these extensions. For simple-modify-headers:

simple-modify-headers settings

There are lots of other extensions/addons out there but I can't name all of them here. Just search "Modify Header extension addon [your browser]" to find one that fits your needs.

  1. Use another library

You could use Selenium Wire. This library might be exactly what you want:

Selenium Wire extends Selenium's Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.

Example:

Install via pip:

pip install selenium-wire

Download and install a driver for your browser: Chrome Driver or Gecko Driver.

Choose a version compatible with your browser.

To get your browser version: on Firefox go to menu > help > about; on Chrome go to menu > about chrome

Install OpenSSL:

# For apt based Linux systems
sudo apt install openssl

See the documentation for more details on the installation.

from seleniumwire import webdriver  # Import from seleniumwire

# Create a new instance of the Chrome driver (or Firefox)
driver = webdriver.Chrome()

# Create a request interceptor
def interceptor(request):
    del request.headers['User-Agent']  # Delete the header first
    request.headers['User-Agent'] = 'Custom User-Agent'

# Set the interceptor on the driver
driver.request_interceptor = interceptor

# All requests will now use 'some_referer' for the referer
driver.get('https://www.bing.com/search?q=TEST123')

I derived the code above from this answer.

You can see Selenium's documentation for other browsers, if you need it.

Maicon Mauricio
  • 2,052
  • 1
  • 13
  • 29
  • Thanks for your answer, I tried your solution however the driver is opening the browser in a weird test environment that doesn't serve my purposes. Let me know if there is a way to get it to open in my actual browser. – Hammaad May 24 '21 at 03:01
  • @Hammaad, I updated my answer. Does the first item meet your needs? – Maicon Mauricio May 24 '21 at 13:25
  • 1
    Your extension idea worked great! I was able to toggle that on manually then run my script which opened the pages with the modified header. – Hammaad May 24 '21 at 18:35