Python web requests: Accessing json data from web response-CodePudding

I have a question with probably a well-known answer. However I couldnt articulate it well enough to find answers on google.

Lets say you are using the developer interface of Chrome browser (Press F12). If you click on the network tab and go to any website, a lot of files will be queried there for example images, stylesheets and JSON-responses.

I want to parse these JSON-responses using python now.

Thanks in advance!

CodePudding user response：

I use requests to get the data, and it comes back as a Python dictionary:

import requests

r = requests.get("url/spotted/with/devtools")
r.json()["keys_observed_in_devtools"]

CodePudding user response：

You can save the network requests to a .har file (JSON format) and analyze that.

In your network tools panel, there is a download button to export as HAR format.

import json
with open('myrequests.har') as f:
    network_data = json.load(f)

print(network_data)

Or, as Jack Deeth answered you can make the requests using Python instead of your browser and get the response JSON data that way.
Though, this can sometimes be difficult depending on the website and nature of the request(s) (for example, needing to login and/or figuring out how to get all the proper arguments to make the request)

CodePudding user response：

Perhaps you can try using Selenium. Maybe the answers on this question can help you.