Use BeautifulSoup to scrape data but I'm not get data all-CodePudding

I want scrape data in table but nothing data after I request url

import requests
url='https://iprice.hk/insights/mapofecommerce/iframe/?lang=en&loc=th'
headers={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'}
r=requests.get(url,headers=headers)
print(r.content.decode())

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.content,'html5lib')

table=soup.find('div', attrs={'id':'data'})

I failed to retrieve data

print(table.prettify())

I am not sure if it's related to data permission or not.

CodePudding user response：

As Scott Hunter says, pulling the data from the csv is a way out to do this:

import requests
import csv 

download = requests.get('https://ipg-moe.s3-ap-southeast-1.amazonaws.com/th/2021-q3.csv')

decoded_content = download.content.decode('utf-8')

cr = csv.reader(decoded_content.splitlines(), delimiter=',')
rows = list(cr)

# first element of rows are column_names, we'll delete it
del rows[0]

for row in rows:
    print(f'''
    Name: {row[1]}
    Url: {row[2]}
    Traffic: {row[4]}
    iOS: {row[5]}
    Android: {row[6]}
    Line: {row[7]}
    Instagram: {row[8]}
    Facebook: {row[9]}
    Type: {row[10]}
    Category: {row[11]}
    Location: {row[12]}
    ''')