Using Selenium how to get network request

Question

I want to take all the network requests using selenium. I am not getting any way to find this solution if anyone can suggest to me or provide a code or library that would be appreciated.

This is my code:

ChromeOptions options = new ChromeOptions();
// Setting some chrome features here

ProxyServer proxyServer = new ProxyServer(4444);
proxyServer.start();

Proxy proxy = proxyServer.seleniumProxy();

DesiredCapabilities capabilities = DesiredCapabilities.chrome();

capabilities.setCapability(ChromeOptions.CAPABILITY, options);
capabilities.setCapability(CapabilityType.PROXY, proxy);

WebDriver driver = new ChromeDriver(capabilities); // Error happens here

are you looking for this: https://www.openhub.net/p/selenium-profiler ? — V. Rob, Aug 24 '17 at 08:34
another way - using proxy for capture all network activity, in fast search I found this: https://github.com/lightbody/browsermob-proxy — V. Rob, Aug 24 '17 at 08:35
example for python https://gist.github.com/mkaz/3047779 and docs https://www.w3.org/TR/navigation-timing/#processing-model — Alexey Shrub, Jul 09 '19 at 15:36
@AlexeyShrub any way to get status code from XHR using python? — jainam shah, Aug 26 '21 at 06:37

Shubham Jain · Accepted Answer · 2022-04-25T14:05:53.743

Not exactly open by dev tools but found some network, performance and other results.

Yes you can do that using JavascriptExecutor

Below Code will give you all performance, network etc entries:-

ChromeOptions options = new ChromeOptions();
options.addArguments("start-maximized");
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
driver.get("http://www.google.com");
String scriptToExecute = "var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return network;";
String netData = ((JavascriptExecutor)driver).executeScript(scriptToExecute).toString();
System.out.println(netData);

OR Below Code will give you specific performance entries:-

DesiredCapabilities d = DesiredCapabilities.chrome();
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
d.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
WebDriver driver = new ChromeDriver(d);
driver.get("https://www.google.co.in/");
driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
LogEntries les = driver.manage().logs().get(LogType.PERFORMANCE);
for (LogEntry le : les) {
    System.out.println(le.getMessage());
}

The first code retrun network return network;" because of this JS tag. You can remove JS code of entity which you don't require

The second code return perfromance

Hope it will help you :)

The output from the first code example isn't valid JSON (it's missing quotation marks, mostly). In order to fix that and get valid JSON output, instead of `return network;` from within the Javascript string, `return JSON.stringify(network);` will output valid JSON with the quotes and all. — George Pantazes, Aug 24 '18 at 21:07
Thanks for sharing @George Pantazes ... I will try and update it soon — Shubham Jain, Aug 28 '18 at 02:35
Another caveat with the Javascript solution: getting the network requests via `performance` entries will only give the network requests at/up to page load and cannot poll for subsequent async/ajax calls. I believe it is the same for the logging solution since it's also logging Performance (although I haven't tried the logging code). @ShubhamJain for my own education, can you confirm if the two snippets both get Performance requests? I think given the limitations of using Performance, I'm going to use [BrowserMobProxy](https://github.com/lightbody/browsermob-proxy) instead. — George Pantazes, Aug 29 '18 at 22:12
I tried to use this method but I think it's not a complete log. Is there any method to have a more complete network log? for example the type of requests, if the responses were successful ... ? — 0xM4x, Oct 29 '19 at 15:40

Norayr Sargsyan · Answer 2 · 2020-09-01T13:32:48.493

6

It's working for me

ChromeOptions options = new ChromeOptions();
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable( LogType.PERFORMANCE, Level.ALL );
options.setCapability( "goog:loggingPrefs", logPrefs );
WebDriverManager.chromedriver().setup();
WebDriver driver = new ChromeDriver(options);
driver.get("http://www.google.com");
    
List<LogEntry> entries = driver.manage().logs().get(LogType.PERFORMANCE).getAll();
System.out.println(entries.size() + " " + LogType.PERFORMANCE + " log entries found");
 for (LogEntry entry : entries) {
   System.out.println(entry.getMessage());
 }

edited Sep 01 '20 at 13:32

answered Feb 04 '20 at 14:59

Norayr Sargsyan

1,737
1
12
26

same answer I have given in 2017, 3 years before this answer. this is the not the right way to get points in StackOverflow .. if you have some better / different code only then share another answer .. dublicate same code make this site unreliable – Shubham Jain Apr 25 '22 at 14:08
Look at the code and then try to run yours and then mine, they are different! – Norayr Sargsyan Apr 27 '22 at 10:34

score 3 · Answer 3 · edited Apr 23 '19 at 12:45

You can use "browsermob-proxy", "LoggingPreferences", "CloseableHttpClient", "HttpURLConnection" for getting the logs
If you wish not to use browser and want to get the response, then I would suggest to go for "CloseableHttpClient".
Copy the URI ("www.somewebsite.com/v1/api/sign-in?"). Get the request payload(which will be available in that particular API URI). Pass all the parameters with "&" like this "www.somewebsite.com/v1/api/sign-in?&username=xyz&password=1234566&app_id=12123214324234134&app_secret=213242345345345" (Remember app id and app secret is very unique and do not expose it anywhere)
Once you get the URI, this code will give you JSON format response

            HttpPost request = new HttpPost(str);
            request.setHeader("content-type", "application/json");
            HttpResponse response = client.execute(request);
            BufferedReader bufReader = new BufferedReader(new InputStreamReader(
                    response.getEntity().getContent()));
            while ((line = bufReader.readLine()) != null) {
                builder=String.valueOf(line);
            }

            System.out.println(builder);
        }

The browsermob proxy hasn't been maintained for a couple years. We just released an actively maintained fork named the [BrowserUp Proxy](https://github.com/browserup/browserup-proxy). It has the same API as the BrowserMob proxy, but also has Java 11 support, HTTP/2 support, updated dependencies, and people caring for it. — ebeland, Apr 26 '19 at 01:50

Using Selenium how to get network request

3 Answers3

Linked