8

According to the selenium documentation, interactions between the webdriver client and a browser is done via JSON Wire Protocol. Basically the client, written in python, ruby, java whatever, sends JSON messages to the web browser and the web browser responds with JSON too.

Is there a way to view/catch/log these JSON messages while running a selenium test?

For example (in Python):

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://google.com')

driver.close()

I want to see what JSON messages are going between the python selenium webdriver client and a browser when I instantiate the driver (in this case Chrome): webdriver.Chrome(), when I'm getting a page: driver.get('http://google.com') and when I'm closing it: driver.close().

FYI, in the #SFSE: Stripping Down Remote WebDriver tutorial, it is done via capturing the network traffic between the local machine where the script is running and the remote selenium server.

I'm tagging the question as Python specific, but really would be happy with any pointers.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • Looking at the [Selenium Webdriver API documentation](http://selenium-python.readthedocs.org/api.html) there doesn't appear to be any accessible way of using the library and having it share the JSON it's generating/receiving. I'd also probably lean towards attempting to capture the network traffic. – ydaetskcoR Sep 12 '14 at 16:17
  • @ydaetskcoR thank you, this is what I'm currently thinking about, though not sure if it's doable if both client and the actual driver are on the local machine. Decided to ask community if there are other options. – alecxe Sep 12 '14 at 16:19
  • It's probably the wrong approach but you could always spin up a VM and put a selenium server on that so you can capture the network traffic over the virtual network layer. – ydaetskcoR Sep 12 '14 at 16:20
  • I think this might be helpful: http://webmasters.stackexchange.com/questions/13339/monitoring-json-requests-sent-received-from-the-browser – Richard Sep 12 '14 at 16:35
  • @Richard thanks, I think the topic is more about browser-internet interactions.. – alecxe Sep 12 '14 at 16:38
  • @ydaetskcoR yeah, strictly speaking, it is an option, the only one for now. Thank you. – alecxe Sep 12 '14 at 16:38
  • @ydaetskcoR posted another option, it shows the half of what I want to see, but it is at least something. – alecxe Sep 12 '14 at 16:56

2 Answers2

6

When you use Chrome you can direct the chromedriver instance that will drive Chrome to log more information than what is available through the logging package. This information includes the commands sent to the browser and the responses it gets. Here's an example:

from selenium import webdriver

driver = webdriver.Chrome(service_log_path="/tmp/log")
driver.get("http://www.google.com")
driver.find_element_by_css_selector("input")
driver.quit()

The code above will output the log to /tmp/log. The part of the log that corresponds to the find_element_... call looks like this:

[2.389][INFO]: COMMAND FindElement {
   "sessionId": "b6707ee92a3261e1dc33a53514490663",
   "using": "css selector",
   "value": "input"
}
[2.389][INFO]: Waiting for pending navigations...
[2.389][INFO]: Done waiting for pending navigations
[2.398][INFO]: Waiting for pending navigations...
[2.398][INFO]: Done waiting for pending navigations
[2.398][INFO]: RESPONSE FindElement {
   "ELEMENT": "0.3367185448296368-1"
}

As far as I know, the commands and responses faithfully represent what is going on between the client and the server. I've submitted bug reports and fixes to the Selenium project on the basis of what I saw in these logs.

Louis
  • 146,715
  • 28
  • 274
  • 320
  • This is really something I've read about but haven't used, looks really informative. Though, this is chrome-specific, but after combining with logging outgoing from the driver http requests, you can almost see the whole picture of the puzzle. Thank you very much. – alecxe Sep 13 '14 at 01:53
  • @alecxe Thanks for the bounty! Glad you appreciated the answer. – Louis Oct 28 '14 at 18:50
5

Found one option that almost fits my needs.

Just piping the logger to the stdout allows to see underlying requests being made:

import logging
import sys

from selenium import webdriver


# pipe logs to stdout
logger = logging.getLogger()
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.NOTSET)

# selenium specific code
driver = webdriver.Chrome()
driver.get('http://google.com')

driver.close()

It prints:

POST http://127.0.0.1:56668/session {"desiredCapabilities": {"platform": "ANY", "browserName": "chrome", "version": "", "javascriptEnabled": true, "chromeOptions": {"args": [], "extensions": []}}}
Finished Request
POST http://127.0.0.1:56668/session/5b6875595143b0b9993ed4f66f1f19fc/url {"url": "http://google.com", "sessionId": "5b6875595143b0b9993ed4f66f1f19fc"}
Finished Request
DELETE http://127.0.0.1:56668/session/5b6875595143b0b9993ed4f66f1f19fc/window {"sessionId": "5b6875595143b0b9993ed4f66f1f19fc"}
Finished Request

I don't see the responses, but this is already a progress.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • That's quite a neat idea. Could you do something similar with requests instead of urllib2? It tends to be a bit more useful – ydaetskcoR Sep 12 '14 at 18:02
  • @ydaetskcoR well, selenium [uses `urllib2` under the hood](https://code.google.com/p/selenium/source/browse/py/selenium/webdriver/remote/remote_connection.py) (on Python2.x), so, there is no need to make the same for `requests`. Also, setting debuglevel was not needed, since it appears that selenium logs the requests with `debug` level. Just setting the appropriate log level is enough. Thanks. – alecxe Sep 12 '14 at 18:23
  • Have you succeeded in getting the responses? – Prayson W. Daniel Nov 13 '17 at 04:41