26

I'm trying to extract the response header of a URL request. When I use firebug to analyze the response output of a URL request, it returns:

Content-Type text/html

However when I use the python code:

urllib2.urlopen(URL).info()

the resulting output returns:

Content-Type: video/x-flv

I am new to python, and to web programming in general; any helpful insight is much appreciated. Also, if more info is needed please let me know.

Thanks in advance for reading this post

Protomen
  • 9,471
  • 9
  • 57
  • 124
looter
  • 261
  • 1
  • 3
  • 3
  • This seems like a duplicate of http://stackoverflow.com/questions/843392/python-get-http-headers-from-urllib-call – Jay Taylor Apr 26 '12 at 20:15

6 Answers6

40

Try to request as Firefox does. You can see the request headers in Firebug, so add them to your request object:

import urllib2

request = urllib2.Request('http://your.tld/...')
request.add_header('User-Agent', 'some fake agent string')
request.add_header('Referer', 'fake referrer')
...
response = urllib2.urlopen(request)
# check content type:
print response.info().getheader('Content-Type')

There's also HTTPCookieProcessor which can make it better, but I don't think you'll need it in most cases. Have a look at python's documentation:

http://docs.python.org/library/urllib2.html

syabro
  • 1,857
  • 2
  • 15
  • 30
qingbo
  • 2,130
  • 16
  • 19
  • 7
    for Python 3: `response.info()["content-type"]` – Janus Troelsen Dec 20 '12 at 14:01
  • Is it completely impossible for a site to check if a request has a fake referrer or not? I'm not lucky with what I try, there's always the error "Invalid referer, won't load xy"... – Nearoo Nov 29 '14 at 13:12
  • Also, if info() doesn't show a row "Referer": Can I conclude that the "fake referer" didn't work? – Nearoo Nov 29 '14 at 13:32
5

Content-Type text/html

Really, like that, without the colon?

If so, that might explain it: it's an invalid header, so it gets ignored, so urllib guesses the content-type instead, by looking at the filename. If the URL happens to have ‘.flv’ at the end, it'll guess the type should be video/x-flv.

bobince
  • 528,062
  • 107
  • 651
  • 834
2

This peculiar discrepancy might be explained by different headers (maybe ones of the accept kind) being sent by the two requests -- can you check that...? Or, if Javascript is running in Firefox (which I assume you're using when you're running firebug?) -- since it's definitely NOT running in the Python case -- "all bets are off", as they say;-).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • hmmm, I'm not too sure how there would be two different headers and also how I would be able to distinguish between both. I'm pretty sure javascript is running in firefox... What would be necessary for me to do within python then? – looter Oct 31 '09 at 06:26
  • @looter, there's no direct way to execute Javascript in Python -- if Javascript's playing a crucial role in determining the final contents of the page, your best bet's automating real browsers instead, e.g. via SeleniumRC. – Alex Martelli Oct 31 '09 at 06:39
  • I'm not sure if Javascript is processing the requests, because when I use the network monitoring in firebug, the response header is also viewable within the HTML view. Like I mentioned in my post, I'm really new to python and web programming/scripting so some of this is going over my head, I'm not sure if I'm being descriptive enough. Thanks for your help so far. – looter Oct 31 '09 at 06:45
1

Keep in mind that a web server can return different results for the same URL based on differences in the request. For example, content-type negotiation: the requestor can specify a list of content-types it will accept, and the server can return different results to try to accomodate different needs.

Also, you may be getting an error page for one of your requests, for example, because it is malformed, or you don't have cookies set that authenticate you properly, etc. Look at the response itself to see what you are getting.

Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
0

according to http://docs.python.org/library/urllib2.html there is only get_header() method and nothing about getheader .

Asking because Your code works fine for 

response.info().getheader('Set cookie')

but once i execute 

response.info().get_header('Set cookie')

i get:

Traceback (most recent call last):
  File "baza.py", line 11, in <module>
    cookie = response.info().get_header('Set-Cookie')
AttributeError: HTTPMessage instance has no attribute 'get_header'

edit: Moreover
response.headers.get('Set-Cookie') works fine as well, not mentioned in urlib2 doc....

Yan Foto
  • 10,850
  • 6
  • 57
  • 88
modzello86
  • 433
  • 7
  • 16
  • get_header() is for the urllib2.Request class. The response class uses getheader() instead, which is an unfortunate mismatch. – Mark E. Haase Dec 27 '12 at 20:56
0

for getting raw data for the headers in python2, a little bit of a hack but it works.

"".join(urllib2.urlopen("http://google.com/").info().__dict__["headers"])

basically "".join(list) will the list of headers, which all include "\n" at the end.

__dict__ is a built in python variable for all dicts, basically you can select a list out of a 2d array with it.

and ofcourse ["headers"] is selecting the list value from the .info() response value dict

hope this helped you learn a few ez python tricks :)

Freak
  • 17
  • 2
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Darrow Hartman May 09 '22 at 17:47