338

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?

charleslparker
  • 1,904
  • 1
  • 21
  • 31

3 Answers3

472

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': 'youremail@domain.example'  # This is another valid field
}

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)
Stephen Ostermiller
  • 23,933
  • 14
  • 88
  • 109
wkl
  • 77,184
  • 16
  • 165
  • 176
  • 7
    You can also access the headers you sent with `response.request.headers` , this works because the original request object is an attribute of the response object. See also [http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects](http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects) – here May 04 '14 at 04:29
  • 5
    The default value is also available as requests.utils.default_user_agent() if you want to just augment that with your own info. – nealmcb Oct 30 '15 at 17:00
  • 3
    It's not correct. It clobbers the rest of the headers. He should get a copy of defaults from requests.utils.default_user_agent() and update it, and send those. – Chad Miller Oct 14 '16 at 17:17
  • 2
    for easyness, on https://httpbin.org/headers (downloadable stuff) you can get the browser headers then make your query appear you – m3nda Dec 23 '16 at 06:53
  • 1
    At least in `2.13.0`, the headers are not clobbered and the [docs](http://docs.python-requests.org/en/master/user/quickstart/#custom-headers) just tell you to use the `headers` kwarg. – Jmills May 31 '17 at 00:02
96

It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})

session.get('https://httpbin.org/headers')

By default, session also manages cookies for you. In case you want to disable that, see this question.

lolzeen
  • 3
  • 4
user
  • 23,260
  • 9
  • 113
  • 101
10

It will send the request like browser

import requests

url = 'https://Your-url'
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}

response= requests.get(url.strip(), headers=headers, timeout=10)
Rajeshkanna Purushothaman
  • 750
  • 5
  • 11