Python: Need to attach extra header, after hidden defaults added by urllib2, before the request is sent

Question

I know how to get at the response headers of a urllib2 request and also how to access those sent and print them out and the request is made, as detailed in the responses to this question.

However, I need to intercept the request after the default headers, such as 'Content-Type' are added by the OpenerDirector, but before, the request is actually sent, because I need to add an extra authorization header (to do with the API I'm working with) which is a hash of various things, including all of the standard headers on the request.

Note also I am using my own subclass of Request, which enables me to send PUT/HEAD/DELETE requests in addition to GET/POST ones and this functionality must be preserved by any solution to this new problem I am having.

TL;DR: Need to access all the default headers added by the OpenerDirector and use them to add a new header before the request is actually sent.

Answer 1

Just some ideas:

The implementation in urllib2.py:

def _open(self, req, data=None):
    result = self._call_chain(self.handle_open, 'default',
                              'default_open', req)
    if result:
        return result

    protocol = req.get_type()
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
    if result:
        return result

    return self._call_chain(self.handle_open, 'unknown',
                            'unknown_open', req)

I would suggest you copy urllib2.py for debugging to your directory, and then add debugging output. You need to look at the dictionary self.handle_open...

Answer 2

Here is a different solution: If you calculate the headers which get generated, these headers are already there and you hash value will be correct. Which headers get generated? I guess content-length is one of them.

Answer 3

In my case I have just decided to manually set all the headers that need to be hashed before the request goes to the OpenerDirector, instead of trying to intercept and do it. I've only been working with Python about a week, so I don't really have the confidence and depth of knowledge to embark on a more complex solution at this time.