9

How can I log all HTTP requests and responses of a page load over Webdriver with PhantomJS? I am using python and my super simple test script looks like this:

from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('http://www.golem.de')

I already found the capabilities in PhantomJS:

page.onResourceRequested = function (request) {
    console.log('Request ' + JSON.stringify(request, undefined, 4));
};

But I don't know how to stick this together with Selenium Webdriver respectively Ghostdriver. How could I do this?

Chris Martin
  • 30,334
  • 10
  • 78
  • 137
Thorben
  • 953
  • 13
  • 28
  • 6
    I recently found out that it is possible to get some HTTP information in HAR format with `driver.get_log("har")`, but i'm interessted in every form of redirect, whose aren't reported in every case. Any suggestions? – Thorben Feb 26 '14 at 14:47

3 Answers3

2

One way to log all network traffic is to use the wonderful tool strace, logging all network requests (and data) to a file.

strace -s9999 -e trace=network curl http://example.com > /dev/null

Partial output:

sendto(3, "GET / HTTP/1.1\r\nUser-Agent: curl/7.32.0\r\nHost: example.com\r\nAccept: */*\r\n\r\n", 75, MSG_NOSIGNAL, NULL, 0) = 75
recvfrom(3, "HTTP/1.1 200 OK\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=604800\r\nContent-Type: text/html\r\nDate: Sun, 08 Ju...
johntellsall
  • 14,394
  • 4
  • 46
  • 40
  • That output is a bit more... low-level... than I was hoping for :( But thanks, much better than nothing. – Chris Martin Jun 08 '14 at 00:36
  • I know this is not an answer to your question but maybe someone else finds it useful. If you need HTTP traffic just for dev purposes, then you could consider temporary use Firefox webdriver so you can check everything. Once everything is correct, you can switch back to phantomjs – serguitus Nov 25 '16 at 16:25
2

Another general low level way, but slightly higher level than strace is tcpdump. You could filter to the specific listening port range and destination host of your server app. You can also log the packets for later analysis if needed. Using the -A (ASCII) dump option you can filter for the request to a given page. A simple example for request to localhost on port 80:

tcpdump -i lo -A -nn dst port 80 and dst host `hostname`

I'm sure Wireshark or similar software could do this type of protocol specific filtering too.

johntellsall
  • 14,394
  • 4
  • 46
  • 40
Francis M. Bacon
  • 665
  • 7
  • 20
2

As Torben said, the driver.get_log("har") is a solution, and I think it is a best solution to me.

from selenium import webdriver

service_args = ['--ignore-ssl-errors=yes']
driver = webdriver.PhantomJS(service_args=service_args)
driver.get('https://www.google.com/')
screenshot = driver.get_screenshot_as_png()
imgname = "google.png"
save_img = open(imgname, 'a')
save_img.write(screenshot)
save_img.close()
print driver.get_log('har')
driver.quit()

For another solutions, we could refer to: 1. browsermob-proxy, 2. Or using webdriver/firebug to catch the network (seems the netexport cannot been verified by Firefox, and maybe we could use the firebug itself, for the newest firebug, it has the feature to export the har) 3. Same as 2, another solution is here: How to capture all requests made by page in webdriver? Is there any alternative to Browsermob?

Ken
  • 325
  • 3
  • 10