Can't get data from site using requests in Python-CodePudding

I'm trying to get text from this

CodePudding user response：

That particular server appears to be gating responses not on the User-Agent, but on the Accept-Encoding settings. You can get a normal response with:

import requests
url = "https://www.spaceweatherlive.com/includes/live-data.php?object=solar_flare&lang=EN"
headers = {
    "Accept-Encoding": "gzip, deflate, br",
}
content = requests.get(url, headers=headers)
print(content.text)

Depending on how the server responds over time, you might need to install the brotli package to allow requests to decompress content compressed with it.

CodePudding user response：

You just need to add user-agent like below.

import requests

url = "https://www.spaceweatherlive.com/includes/live-data.php?object=solar_flare&lang=EN"

payload={}
headers = {
    'User-Agent': 'PostmanRuntime/7.29.0',
    'Accept': '*/*',
    'Cache-Control': 'no-cache',
    'Host': 'www.spaceweatherlive.com',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive'
 }
response = requests.get(url, headers=headers)
print(response.text)