Get headers sent in urllib2 http request

Question

I know how to get the headers RECEIVED

resp = urllib2.urlopen('http://www.google.com')
print resp.info()

But how do you access the headers SENT to 'http://www.google.com' ?

I usually use wireshark to analyze and see what is actually being sent, but I'd like to have access to this information in my script.

See here for how to check the actual request: http://stackoverflow.com/questions/603856/how-do-you-get-default-headers-in-a-urllib2-request — onteria_, May 22 '11 at 02:55

score 2 · Answer 1 · answered May 22 '11 at 03:34

2

import httplib
import urllib2

class CustomHTTPConnection(httplib.HTTPConnection):
    def request(self, method, url, body=None, headers={}):
        print headers
        self._send_request(method, url, body, headers)

class CustomHTTPHandler(urllib2.AbstractHTTPHandler):
    def http_open(self, req):
        return self.do_open(CustomHTTPConnection, req)
    http_request = urllib2.AbstractHTTPHandler.do_request_

if __name__ == '__main__':
    opener = urllib2.OpenerDirector()
    opener.add_handler(CustomHTTPHandler())
    res = opener.open('http://www.google.it/')

answered May 22 '11 at 03:34

sherpya

4,890
2
34
50

How do I access the same headers outside of the class? I need to do something with the headers besides just print it out. – Raj Oct 05 '11 at 18:25
unfortunately there isn't an easy way without using globals since opener.open returns an addinfourl made with HTTPMessage of HTTPResponse (this may say nothing to you) anyway you can append headers to response and maybe prefix with req_ I'm adding the code as another answer or there is a way to format in comments? – sherpya Nov 03 '11 at 00:24

score 0 · Answer 2 · answered Nov 03 '11 at 00:30

import httplib
import urllib2

class CustomHTTPConnection(httplib.HTTPConnection):
    def request(self, method, url, body=None, headers={}):
        self.req_headers = headers
        self._send_request(method, url, body, headers)
    def getresponse(self, buffering=False):
        resp = httplib.HTTPConnection.getresponse(self, buffering)
        for key, value in self.req_headers.items():
            resp.msg.headers.append('req_%s: %s\r\n' % (key, value))
        return resp

class CustomHTTPHandler(urllib2.AbstractHTTPHandler):
    def http_open(self, req):
        resp = self.do_open(CustomHTTPConnection, req)
        return resp
    http_request = urllib2.AbstractHTTPHandler.do_request_

if __name__ == '__main__':
    opener = urllib2.OpenerDirector()
    opener.add_handler(CustomHTTPHandler())
    res = opener.open('http://www.google.it/')
    info = res.info()
    print info

Get headers sent in urllib2 http request

2 Answers2