2

I have a question with probably a well-known answer. However I couldnt articulate it well enough to find answers on google.

Lets say you are using the developer interface of Chrome browser (Press F12). If you click on the network tab and go to any website, a lot of files will be queried there for example images, stylesheets and JSON-responses.

I want to parse these JSON-responses using python now.

Thanks in advance!

sytech
  • 29,298
  • 3
  • 45
  • 86
Michel1506
  • 73
  • 1
  • 7
  • Right click the json request in the network manager, copy the CURL request and convert it to a python request with https://curlconverter.com/. Quite easy workflow, I use it all the time. – RJ Adriaansen Feb 07 '22 at 20:31
  • 1
    "*Do you know a python library that might be helpful?*" Questions seeking library recommendations are explicitly off-topic here per the scope of the site defined in the [help/on-topic]. – esqew Feb 07 '22 at 20:32
  • A quick Google search for `parse JSON data python` yields several threads on this very site, including this duplicate: [How to parse data in JSON format?](https://stackoverflow.com/questions/7771011/how-to-parse-data-in-json-format) The ability to parse standard-compliant JSON is built-in to the Python language, and does not require any third-party libraries as you seem to think in your question – esqew Feb 07 '22 at 20:32

3 Answers3

2

I use requests to get the data, and it comes back as a Python dictionary:

import requests

r = requests.get("url/spotted/with/devtools")
r.json()["keys_observed_in_devtools"]
Jack Deeth
  • 3,062
  • 3
  • 24
  • 39
2

You can save the network requests to a .har file (JSON format) and analyze that.

In your network tools panel, there is a download button to export as HAR format.

export HAR button

import json
with open('myrequests.har') as f:
    network_data = json.load(f)

print(network_data)

Or, as Jack Deeth answered you can make the requests using Python instead of your browser and get the response JSON data that way.
Though, this can sometimes be difficult depending on the website and nature of the request(s) (for example, needing to login and/or figuring out how to get all the proper arguments to make the request)

sytech
  • 29,298
  • 3
  • 45
  • 86
  • Thanks for your comment. However your solution needs manual preparation and Id like to do it without :) – Michel1506 Feb 07 '22 at 22:55
  • I did not know about `.har` files but you bet I'll be using these going forward. – Jack Deeth Feb 07 '22 at 22:58
  • @Michel1506 understood. It is possible to capture the HAR data dynamically from a browser (optionally controlled by selenium) using BrowserMob proxy. Though, using `requests` is probably the better solution, if that's practical. – sytech Feb 08 '22 at 00:22
-1

Perhaps you can try using Selenium. Maybe the answers on this question can help you.

Florentin Alexandru Iftimie
  • 89
  • 1
  • 5