How to get URLs from the Chrome DevTools Network tab?

Question

Given a certain website containing many resources, I need to automate the process of getting all the resources' URLs. To complicate matters, these URLs are not contained in the initially loaded markup but are instead inserted into the DOM via JavaScript, based on user interaction with the page.

Therefore I must retrieve results from the Network tab of Chrome's DevTools. But I'm having difficulty getting started.

Here's my first attempt:

Imports System.Text
Imports OpenQA.Selenium
Imports OpenQA.Selenium.Chrome
Imports OpenQA.Selenium.Support.UI

Friend Module Main
  Public Sub Main()
    Dim oBuilder As StringBuilder
    Dim oOptions As ChromeOptions
    Dim oDriver As IWebDriver
    Dim oWait As WebDriverWait
    Dim sType As String

    sType = LogType.Browser

    oBuilder = New StringBuilder

    oOptions = New ChromeOptions
    oOptions.SetLoggingPreference(sType, LogLevel.All)

    oDriver = New ChromeDriver(oOptions)
    oDriver.Navigate.GoToUrl("http://example.com")

    oWait = New WebDriverWait(oDriver, TimeSpan.FromSeconds(15))
    oWait.Until(Function(Driver) Driver.FindElement(By.TagName("a")))

    oDriver.Manage.Logs.GetLog(sType).ToList.ForEach(Sub(Log)
                                                       oBuilder.AppendLine($"Level:   {Log.Level}")
                                                       oBuilder.AppendLine($"Message: {Log.Message}")
                                                     End Sub)

    Console.WriteLine(oBuilder.ToString)
  End Sub
End Module

Upon the first run of this code, the StringBuilder contained only one LogEntry:

Timestamp            Level  Message
---------            -----  -------
2/25/2019 5:05:05 PM Severe http://example.com/favicon.ico - Failed to load resource: the server responded with a status of 404 (Not Found)

Since that first run, however, no logs are retrieved. Moreover, this is not the log I need. I need resource URLs.

There are three main problems to overcome here:

1. When a page is retrieved from the browser's local cache, it appears there is no output to the log
2. There doesn't appear to be a way to set the LogLevel, even though my code attempts to do so early on
3. These logs are not resource URLs

How can I get the URLs from the DevTools Network tab? I found this quick sample—in fact it inspired my code above—but it's using the Java SDK. The two APIs seem slightly different.

Answer 1

According to How to set Chrome preferences using Selenium Webdriver .NET binding? you will need your own class like

public class ChromeOptionsWithPrefs: ChromeOptions
{
    public Dictionary<string,object> prefs { get; set; }
}

public static void Initialize()
{
    var options = new ChromeOptionsWithPrefs();
    options.prefs = new Dictionary<string, object>
    {
        { "enableNetwork", true },
        { "traceCategories", "netlog,devtools.timeline,devtools" }
    };
    _driver = new ChromeDriver(@"C:\path\chromedriver", options);
}

I cannot verify the code, but in Java you need to enable network logging just like that. The trace categories can be found with the pseudo url chrome://tracing/ in Chrome.

Answer 2

You can get all resources with Selenium using browser logs.

 def get_logs(self):
        logs = self.browser.get_log('performance')
        return logs