How to scrape SVG element from a website using Beautiful Soup?-CodePudding

from bs4 import BeautifulSoup
import requests
import random

id_url = "https://codeforces.com/profile/akash77"
id_headers = {
    "User-Agent": 'Mozilla/5.0(Windows NT 6.1Win64x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 87.0 .4280 .141 Safari / 537.36 '}
id_page = requests.get(id_url, headers=id_headers)
id_soup = BeautifulSoup(id_page.content, 'html.parser')

id_soup = id_soup.find('svg')
print(id_soup)

I'm getting None as the output for this.

If I parse the <div> element in which this <svg> tag is contained, the contents of the <div> element are not getting printed. The find() works for all HTML tags except the SVG tag.

CodePudding user response：

The webpage is rendered dynamically with Javascript, so you will need

Sometimes, you need to run a browser in headless mode (without opening a chrome UI), for that you can pass a 'headless' option to the driver.

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('headless')

# then pass options to the driver

driver = webdriver.Chrome(service=s, options=options)

CodePudding user response：

svg tag is not included in the source code, it is rendered by Javascript.