How do I set HTTP headers using Python's urllib?

Question

I am pretty new to Python's urllib. What I need to do is set a custom HTTP header for the request being sent to the server.

Specifically, I need to set the Content-Type and Authorization HTTP headers. I have looked into the Python documentation, but I haven't been able to find it.

score 132 · Answer 1 · edited Mar 28 '19 at 09:00

132

For both Python 3 and Python 2, this works:

try:
    from urllib.request import Request, urlopen  # Python 3
except ImportError:
    from urllib2 import Request, urlopen  # Python 2

req = Request('http://api.company.com/items/details?country=US&language=en')
req.add_header('apikey', 'xxx')
content = urlopen(req).read()

print(content)

edited Mar 28 '19 at 09:00

Kir Chou

2,980
1
36
48

answered Jul 21 '14 at 16:37

Cees Timmerman

17,623
11
91
124

Can we do the same thing with requests q.add_header('apikey', 'xxx') – user3378649 Jun 14 '15 at 17:41
What do you mean, @user3378649? – Cees Timmerman Jun 14 '15 at 22:04
2

@user3378649 may be you means use `requests` python package [custom headers](http://docs.python-requests.org/en/master/user/quickstart/#custom-headers) – WeizhongTu Apr 15 '16 at 01:27
2

THIS answer - a thousand times YES (thanks!). I have been struggling for hours trying to find a common interface for python 2 and 3 (between urllib, urllib2 and urllib3). – Beorn Harris Sep 20 '19 at 08:18

score 103 · Accepted Answer · edited Sep 16 '17 at 14:29

103

adding HTTP headers using urllib2:

from the docs:

import urllib2
req = urllib2.Request('http://www.example.com/')
req.add_header('Referer', 'http://www.python.org/')
resp = urllib2.urlopen(req)
content = resp.read()

edited Sep 16 '17 at 14:29

Lars Blumberg

19,326
11
90
127

answered Oct 28 '11 at 18:53

Corey Goldberg

59,062
28
129
143

score 22 · Answer 3 · answered Oct 28 '11 at 18:44

Use urllib2 and create a Request object which you then hand to urlopen. http://docs.python.org/library/urllib2.html

I dont really use the "old" urllib anymore.

req = urllib2.Request("http://google.com", None, {'User-agent' : 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'})
response = urllib2.urlopen(req).read()

untested....

score 1 · Answer 4 · edited Dec 20 '15 at 15:07

1

For multiple headers do as follow:

import urllib2
req = urllib2.Request('http://www.example.com/')
req.add_header('param1', '212212')
req.add_header('param2', '12345678')
req.add_header('other_param1', 'sample')
req.add_header('other_param2', 'sample1111')
req.add_header('and_any_other_parame', 'testttt')
resp = urllib2.urlopen(req)
content = resp.read()

edited Dec 20 '15 at 15:07

rkosegi

14,165
5
50
83

answered May 26 '15 at 08:39

Gil Allen

1,169
14
24

1

Do _not_ do this, simply pass in a headers dictionary if you have multiple header fields to use. – Raleigh L. Jun 08 '22 at 08:31
1

guess you are right - but hey, this was 2015 for Python 2.7 Theses days - I'm populating a dictionary. – Gil Allen Jun 09 '22 at 09:10

How do I set HTTP headers using Python's urllib?

4 Answers4

Linked