-2

In Chrome, you can go to developer tools > network tab to see all the requests the website is making.

What would be a good way to get the list of these requests programmatically? I guess I can grab the content of the site, grab all the URLs that are in the page and parse them, but that seems a bit tedious, especially if the requests are being made from a JS file.

Is there an easier way?

ericbae
  • 9,604
  • 25
  • 75
  • 108
  • The network tab does _not_ show you the requests the website is making, it shows you the requests the _browser_ is making. This is a very important distinction. If the browser requests are indeed what you're looking for, you can probably accomplish that via [Puppeteer](https://stackoverflow.com/questions/52969381/how-can-i-capture-all-network-requests-and-full-response-data-when-loading-a-pag) or some similar headless browser based solution that will execute javascript. – Alex Howansky Mar 14 '22 at 20:44

1 Answers1

0

On the server side, there's the access log that will have request information for all requests made against it. It might take some configuration to save all the fields you want to use (method, POST data, cookies, time, remote client address) but this is more efficient than trying to do things from a browser. Each web server has a different way of configuring its log files.

Xavier J
  • 4,326
  • 1
  • 14
  • 25