I want to obtain on a single DataFrame all the different tables from each company scraping this web page:

https://rk.americaeconomia.com/display/embed/500-latam/2021

or

https://www.americaeconomia.com/negocios-industrias/estas-son-las-500-mayores-empresas-de-america-latina-2021

It has been very difficult to find a solution.

CodePudding user response：

Situation

There are two different types of <tr> one with data and another with extra data corresponding to the first one we have to bring them together. Another point is that extra data is not stored in common table structure

How to achieve?

Each combination of data and extra data has to be joined to a single row - There are many options to do so, this approach deals with dataframes. Joined dataframes are stored in a list and would be concatenated after iterating has finished.

data = []
for row in soup.select('#awesomeTable tbody tr.extraDataRow'):

    df = pd.DataFrame([list(row.find_previous('tr').stripped_strings)], columns=list(soup.select_one('#awesomeTable tr').stripped_strings))
    df = df.join(pd.DataFrame(dict([list(x.stripped_strings) for x in row.select('p')]), index=df.index))
    data.append(df)


df = pd.concat(data)

Example

import requests
import pandas as pd
from bs4 import BeautifulSoup
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}
r = requests.get('https://rk.americaeconomia.com/display/embed/500-latam/2021',headers=headers)
soup = BeautifulSoup(r.text,'lxml')
data = []
for row in soup.select('#awesomeTable tbody tr.extraDataRow'):
   
    df = pd.DataFrame([list(row.find_previous('tr').stripped_strings)], columns=list(soup.select_one('#awesomeTable tr').stripped_strings))
    df = df.join(pd.DataFrame(dict([list(x.stripped_strings) for x in row.select('p')]), index=df.index))
    data.append(df)


df = pd.concat(data)

Output