I'm trying to extract data from a website with BeautifulSoup. I'm actually stuck with this :
"Trad. de l'anglais par < a href="/searchinternet/advanced?all_authors_id=35534&SearchAction=1">Camille Fabien < /a>"
I want to get the names of translaters but the tag uses their id.

my code is

translater = soup.find_all("a", href="/searchinternet/advanced?all_authors_id=")

I tried with a str.startswith but it doesn't work. Can someone help me plz?

CodePudding user response：

Providing your HTML is correct, static (doesn't get loaded with javascript after initial page load), this is one way to select that/those links:

from bs4 import BeautifulSoup as bs

html = '''<p>Trad. de l'anglais par <a href="/searchinternet/advanced?all_authors_id=35534&SearchAction=1">Camille Fabien </a></p>'''

soup = bs(html, 'html.parser')
a = soup.select('a[href^="/searchinternet/advanced?all_authors_id="]')
print(a[0])
print(a[0].get_text(strip=True))
print(a[0].get('href'))

Result in terminal:

<a href="/searchinternet/advanced?all_authors_id=35534&amp;SearchAction=1">Camille Fabien </a>
Camille Fabien
/searchinternet/advanced?all_authors_id=35534&SearchAction=1

EDIT: Who doesn't like a challenge?... Based on further comments made by OP, here is a way of obtaining titles, authors, translators and illustrator from that page - considering there can be one, or more translators/one or more illustrators:

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

headers = {
    'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
    }

url = 'https://www.gallimard.fr/searchinternet/advanced/(editor_brand_id)/1/(fserie)/FOLIO-JUNIOR LIVRE HEROS::Folio Junior - Un Livre dont Vous êtes le Héros @ DEFIS FANTASTIQ::Série Défis Fantastiques/(limit)/3?date[from]=1980-01-01&date[to]=1995-01-01&SearchAction=OK'

big_list = []
r = requests.get(url, headers=headers)
soup = bs(r.text, 'html.parser')
items = soup.select('div[] > table div[]')
print()
for i in items:
    title = i.select_one('div[] h3')
    author = i.select_one('div[] a')
    history = i.select_one('p[]')
    translators = [[y.get_text() for y in x.find_previous_siblings('a')] for x in history.contents  if "Illustrations" in x]
    illustrators = [[y.get_text() for y in x.find_next_siblings('a')] for x in history.contents  if "Illustrations" in x]
    big_list.append((title.text.strip(), author.text.strip(), ', '.join([x for y in translators for x in y]), ', '.join([x for y in illustrators for x in y])))
df = pd.DataFrame(big_list, columns = ['Title', 'Author', 'Translator(s)', 'Illustrator(s)'])
print(df)

Result in terminal:

Bear in mind this method fails for Le Manoir de l'Enfer, because word 'Illustrations' is not found in text. It's down to the OP to find a solution for that one.

BeautifulSoup documentation can be found at https://beautiful-soup-4.readthedocs.io/en/latest/index.html

Also, Pandas docs can be found here: https://pandas.pydata.org/pandas-docs/stable/index.html

CodePudding user response：

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("./test.html", "r"),'html.parser') #returns a list

names = []

for elem in soup:
    names.append(elem.text)