I am trying to scrape data from all those list items and also from featherlight container. Code I've written to extract list items is as follows

driver.get('https://www.novitecgroup.com/en/brands/ferrari/roma/')
actions = ActionChains(driver)
parts_list = driver.find_element_by_class_name('tuning-parts-categories__content')
parts_list_element = parts_list.find_elements_by_tag_name('div')
temp = parts_list_element[0].find_elements_by_tag_name('li')
print(temp)

but temp returns as a empty list. neither am I able to get data from featherlight. Thanks in advance for your help.

1> Data within the featherlight to be extracted

2> webpage and respective source code I am trying to extract data from

3> New Error

CodePudding user response：

wait=WebDriverWait(driver,60)                                 
driver.get('https://www.novitecgroup.com/en/brands/ferrari/roma/')

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#cookie-confirm__submit"))).click()
parts_lists = wait.until(EC.visibility_of_all_elements_located((By.XPATH,"//div[@class='tuning-parts-categories__content']/div[1]//li/a[1]")))
for x in parts_lists:
    driver.execute_script("arguments[0].scrollIntoView();", x)
    time.sleep(2)
    driver.execute_script("arguments[0].click();", x)
    try:
        featherlight=WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div.tuning-part-popup__text")))
        product=featherlight.find_elements(By.XPATH,".//p[1]")
        if len(product) == 0:
            product = 'Empty'
        else:
            product = product[0].text
        
        box=featherlight.find_elements(By.XPATH,".//p[2]")
        if len(box) == 0:
            torque = 'Empty'
            performance = 'Empty'
        else:
            torque = box[0].text.split("\n")[1]
            performance = box[0].text.split("\n")[0]
        print(product,performance,torque)
    except Exception as e:
        print(str(e))
        pass
    close=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"button.featherlight-close-icon.featherlight-close")))
    driver.execute_script("arguments[0].click();", close)

Imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

Outputs:

NOVITEC N-TRONIC is a processor-controlled device to be connected to the original engine management with the Delivered Plug&Play wiring harness. The Kit includes two NOVITEC N-TRONIC modules. It is a ready to mount Plug & Play System and it is easy to install. Performance: 506 kW (688 hp) / 7.400 rpm Max. torque: 851 Nm / 3.750 rpm
NOVITEC N-TRONIC Kit, X-pipe system, catalyst-replacement pipe, SWITCHTRONIC and TECTRONIC Performance: 518 kW (704 hp) / 7.400 rpm Max. torque: 882 Nm / 3.750 rpm
NOVITEC N-TRONIC Kit, X-pipe system, sport metal catalysts, TECTRONIC and SWITCHTRONIC Performance: 514 kW (699hp) / 7.450 rpm         Max. torque: 877 Nm / 3.700 rpm
to use with NOVITEC exhaust and original exhaust system. Empty Empty
to use with NOVITEC- and original exhaust system (-4kg), 100 Zeller sport metal catalysts with reduced exhaust back pressure complete covered with high-temperature heat protection. Empty Empty
Message: 

Message: 

To use with NOVITEC sport metal catalysts or NOVITEC cat replacement pipes to avoid error indications from the exhaust system  In combination with order of a NOVITEC exhaust system Article no. F1 333 80
to use for exhaust systems with flap-regulation to open and close the flaps from the cockpit by Radio Empty Empty