How to grab headers in python selenium-webdriver

Question

I am trying to grab the headers in selenium webdriver. Something similar to the following:

>>> import requests
>>> res=requests.get('http://google.com')
>>> print res.headers

I need to use the Chrome webdriver because it supports flash and some other things that I need to test a web page. Here is what I have so far in Selenium:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://login.comcast.net/login?r=comcast.net&s=oauth&continue=https%3A%2F%2Flogin.comcast.net%2Foauth%2Fauthorize%3Fclient_id%3Dxtv-account-selector%26redirect_uri%3Dhttps%3A%2F%2Fxtv-pil.xfinity.com%2Fxtv-authn%2Fxfinity-cb%26response_type%3Dcode%26scope%3Dopenid%2520https%3A%2F%2Flogin.comcast.net%2Fapi%2Flogin%26state%3Dhttps%3A%2F%2Ftv.xfinity.com%2Fpartner-success.html%26prompt%3Dlogin%26response%3D1&reqId=18737431-624b-44cb-adf0-2a85d91bd662&forceAuthn=1&client_id=xtv-account-selector')
driver.find_element_by_css_selector('#user').send_keys('XY@comcast.net')
driver.find_element_by_css_selector('#passwd').send_keys('XXY')
driver.find_element_by_css_selector('#passwd').submit()
print driver.headers ### How to do this?

I have seen some other answers that recommend running an entire selenium server to get this information (https://github.com/derekargueta/selenium-profiler). How would I get it using something similar to the above with Webdriver?

Could you please elaborate what headers do you want to extract and what for? Thanks. — alecxe, Oct 05 '16 at 20:05

elethan · Accepted Answer · 2016-10-05T21:02:15.757

14

Unfortunately, you cannot get this information from the Selenium webdriver, nor will you be able to any time in the near future it seems. An excerpt from a very long conversation on the subject:

This feature isn't going to happen.

The gist of the main reason being, from what I gather from the discussion, that the webdriver is meant for "driving the browser", and extending the API beyond that primary goal will, in the opinion of the developers, cause the overall quality and reliability of the API to suffer.

One potential workaround that I have seen suggested in a number of places, including the conversation linked above, is to use BrowserMob Proxy, which can be used to capture HTTP content, and can be used with selenium - though the linked example does not use the Python selenium API. It does seem that there is a Python wrapper for BrowserMob Proxy, but I cannot vouch for it's efficacy since I have never used it.

edited Oct 05 '16 at 21:02

answered Oct 05 '16 at 20:17

elethan

16,408
8
64
87

1

what about executing javascript or something within the page to log it to console or something? Is there a (hackish) way to do something like that? – David542 Oct 05 '16 at 20:42
One suggestion I have seen repeatedly on this subject is to use BrowserMob Proxy: https://github.com/lightbody/browsermob-proxy, which can be used with selenium: https://github.com/lightbody/browsermob-proxy#using-with-selenium. However, I do not have experience with this utility. Sorry I couldn't be of more help! – elethan Oct 05 '16 at 20:57
@David542 also, see the last paragraph of my updated answer. It includes a link to a Python wrapper for BrowserMob Proxy, which may work for your use case. – elethan Oct 05 '16 at 21:03

score 7 · Answer 2 · answered Sep 01 '19 at 07:24

7

Now, it is very easy i suppose https://pypi.org/project/selenium-wire/ it is an extension of selenium. use from seleniumwire import webdriver and proceed as usual.

answered Sep 01 '19 at 07:24

munish

4,505
14
53
83

I wish this was more popular & updates still being rolled out, it's so intuitive ... – Heartthrob_Rob Jul 18 '20 at 23:49

score 5 · Answer 3 · edited May 14 '18 at 18:15

5

You could try Mobilenium, a python package (still in development) that binds BrowserMob Proxy and Selenium.

An usage example:

>>> from mobilenium import mobidriver
>>>
>>> browsermob_path = 'path/to/browsermob-proxy'
>>> mob = mobidriver.Firefox(browsermob_binary=browsermob_path)
>>> mob.get('http://python-requests.org')
301
>>> mob.response['redirectURL']
'http://docs.python-requests.org'
>>> mob.headers['Content-Type']
'application/json; charset=utf8'
>>> mob.title
'Requests: HTTP for Humans \u2014 Requests 2.13.0 documentation'
>>> mob.find_elements_by_tag_name('strong')[1].text
'Behold, the power of Requests'

edited May 14 '18 at 18:15

parik

2,313
12
39
67

answered Mar 16 '17 at 11:58

Rafael Ribeiro

311
3
7

i get an error `ImportError: cannot import name 'mobidriver'`. Any ideas how I can fix this? – West Jun 22 '18 at 19:23
@west `pip install mobilenium` – deweydb Apr 24 '20 at 23:02
@West i was wrong, there is an issue with that repo, you can work around it with: `pip install -U git+https://github.com/rafpyprog/Mobilenium` – deweydb Apr 24 '20 at 23:28

score 2 · Answer 4 · answered Dec 21 '17 at 17:02

2

You can get the header via the log (source from Mma's answer)

from selenium import webdriver
import json
driver = webdriver.PhantomJS(executable_path=r"your_path")
har = json.loads(driver.get_log('har')[0]['message']) # get the log
print('headers: ', har['log']['entries'][0]['request']['headers'])

answered Dec 21 '17 at 17:02

J. Does

785
3
10
23

Where do you put the site url? – West Jun 22 '18 at 17:58

score 0 · Answer 5 · answered Apr 18 '20 at 22:10

You can use the JAVASCRIPT built-in method.

It only can be done once the driver has already been created though.

from selenium import webdriver
driver = webdriver.Chrome()
# Store it in a variable and print the value
agent = driver.execute_script("return navigator.userAgent")
print(agent)
# directly print the value
print(driver.execute_script("return navigator.userAgent"))

How to grab headers in python selenium-webdriver

5 Answers5

Linked