I am trying to pull data from this website, but I am getting a Response 403 error when running session.post. Please see code below for reference. Any help would be appreciated.

import requests
from bs4 import BeautifulSoup
import re

username = 'username'
password = 'password'
scrape_url = 'https://app.mapro.us/en/manage/owners/houses'

login_url = 'https://app.mapro.us/en/login'
login_info = {'login': username, 'pwd': password}
headers = {
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
            'Accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Connection': 'keep-alive'
          }

#Start session.
session = requests.session()

#Login using your authentication information.
p = session.post(url=login_url, data=login_info, headers=headers)

print(p)

CodePudding user response：

I don't have account to test with correct login and password but there are some differences when I check it in browser (in DevTools in Firefox/Chrome in tab Network)

The main difference is:

• it sends POST to address https://app.mapro.us/ajax?login=

If I use this link then I get 200 with JSON data

{
    "status": 0,
    "msg": "Authorization denied."
}

Maybe if I would have account then it would give different message.

There are other differences which may be important or not

• it sends POST as AJAX so it has header

  X-Requested-With': 'XMLHttpRequest
  

• it expects response with JSON so it has different header Accept

  'Accept': 'application/json, text/javascript, */*; q=0.01'
  

• it sends POST with cookie SID which it gets in previous GET - so you may need to run session.get('https://app.mapro.us/en/login', ....) before POST

BTW: this GET in browser always get 403 so it seems not important.

import requests

session = requests.session()

# --- GET ---

headers = {
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
            'Accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Connection': 'keep-alive',
          }

url_get = 'https://app.mapro.us/en/login'

p = session.get(url_get, headers=headers)

print(p)
#print(p.text)
print('Cookies SID:', session.cookies.get('SID'))

# --- POST ---

username = 'username'
password = 'password'

login_info = {'login': username, 'pwd': password}

headers = {
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
            'Accept': 'application/json, text/javascript, */*; q=0.01',  # expect JSON 
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Connection': 'keep-alive',
            'X-Requested-With': 'XMLHttpRequest',  # send AJAX
          }

url_post = 'https://app.mapro.us/ajax?login='

p = session.post(url_post, headers=headers, data=login_info)

print(p)
print(p.text)