Selenium (python): not retrieving text in HTML displayed on next pages-CodePudding

I am attempting to scrape the names of drugs, makers and auth.codes (AIC) from our national drug agency.

The code below almost works but only captures the text of what is displayed on the web page but not the text that I see in the HTML but would be displayed on subsequent pages:

So in the following example search I print the drugs up to the ABACUS drug name but then print empty lines. What am I doing wrong? Thank you in advance.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome(
    "/Users/bob/Documents/work/AIFA/scraper/scrape_gu/chromedriver"
)

# start from AIFA's search page and automate steps to drug search function
driver.get("https://farmaci.agenziafarmaco.gov.it/bancadatifarmaci/cerca-farmaco")
lettocompreso = driver.find_element_by_id("conf")
lettocompreso.click()
accetta = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "/html/body/div[5]/div[3]/div/button"))
)
accetta.click()

# now click on the search button with drug (fragment) name
cerca_textbox = driver.find_element_by_id("search")
cerca_textbox.send_keys("AB")
cerca_textbox.send_keys(Keys.ENTER)
# now parse the results
toplistafarmaci = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//*[@id="ul_farm_results"]/li'))
)

# now all data I need are in <span> triples
drugs = []

for drug in driver.find_elements_by_tag_name("span"):
    print(drug.text)

of course you can completely ignore the code until the "now parse the results" comment line which are only there to jump to the actual search.

CodePudding user response：

If you just want to extract the data from all the pages then you will have make the below changes, such as use this xpath //*[@id='ul_farm_results']/li[@style='display: list-item;'] and store everything in a list and then iterate the list on all the pages.

cerca_textbox = driver.find_element_by_id("search")
cerca_textbox.send_keys("AB")
cerca_textbox.send_keys(Keys.ENTER)
# now parse the results
# now all data I need are in <span> triples
drugs = []

wait = WebDriverWait(driver, 30)
total_number_of_pages = 7

while total_number_of_pages > 0:
    try:
        toplistafarmaci = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='ul_farm_results']/li[@style='display: list-item;']")))
        for drug in toplistafarmaci:
            print(drug.text)
            #drugs.append(drug.text)
        wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='>']"))).click()
        time.sleep(2)
        print('Clicked on next page')
        total_number_of_pages = total_number_of_pages - 1
    except:
        print('Could not click on next page arrow')
        break

output:

AIC: 045359
ABACAVIR E LAMIVUDINA ACCORD
Azienda: ACCORD HEALTHCARE, S.L.U.
AIC: 045522
ABACAVIR E LAMIVUDINA ARISTO
Azienda: ARISTO PHARMA GMBH
AIC: 044669
ABACAVIR E LAMIVUDINA AUROBINDO
Azienda: AUROBINDO PHARMA (ITALIA) S.R.L.
AIC: 044114
ABACAVIR E LAMIVUDINA DOC GENERICI
Azienda: DOC GENERICI SRL
AIC: 044980
ABACAVIR E LAMIVUDINA DR. REDDY'S
Azienda: DR. REDDY'S S.R.L.
AIC: 044719
ABACAVIR E LAMIVUDINA EG
Azienda: EG S.P.A.
AIC: 043972
ABACAVIR E LAMIVUDINA LUPIN
Azienda: LUPIN (EUROPE) LIMITED
AIC: 043971
ABACAVIR E LAMIVUDINA MYLAN
Azienda: MYLAN S.P.A.
AIC: 044773
ABACAVIR E LAMIVUDINA MYLAN PHARMA
Azienda: MYLAN S.P.A.
AIC: 044838
ABACAVIR E LAMIVUDINA SANDOZ
Azienda: SANDOZ S.P.A.
AIC: 045348
ABACAVIR E LAMIVUDINA SUN
Azienda: SUN PHARMACEUTICAL INDUSTRIES (EUROPE) B.V.
AIC: 044286
ABACAVIR E LAMIVUDINA TEVA
Azienda: TEVA ITALIA S.R.L.
AIC: 045021
ABACAVIR E LAMIVUDINA ZENTIVA
Azienda: ZENTIVA ITALIA S.R.L.
AIC: 045354
ABACAVIR MYLAN
Azienda: MYLAN S.P.A.
AIC: 042645
ABACAVIR/LAMIVUDINA/ZIDOVUDINA MYLAN
Azienda: MYLAN S.P.A.
AIC: 028508
ABACUS
Azienda: DOMPE' FARMACEUTICI S.P.A.
Clicked on next page
AIC: 022773
ABADOX
Azienda: LA FAR SRL
AIC: 025605
ABAPRIM
Azienda: ISTITUTO GENTILI S.P.A.
AIC: 043658
ABASAGLAR
Azienda: ELI LILLY NEDERLAND B.V.
AIC: 036816
ABBA
Azienda: FIDIA FARMACEUTICI S.P.A.
AIC: 033002
ABELCET COMPLESSO LIPIDICO
Azienda: TEVA ITALIA S.R.L.
AIC: 044868
ABERIPRA
Azienda: ECUPHARMA S.R.L.
AIC: 040915
ABESART
Azienda: GENETIC S.P.A.
AIC: 024480
ABETOL
Azienda: LABORATORIO FARMACEUTICO C.T. S.R.L.
AIC: 049452
ABEVMY
Azienda: MYLAN IRE HEALTHCARE LIMITED
AIC: 026465
ABEX
Azienda: GLORIA MED PHARMA S.R.L.
AIC: 004097
ABIDEC
Azienda: TEOFARMA S.R.L.
AIC: 039371
ABIDOL
Azienda: AURORA LICENSING SRL
AIC: 047244
ABIES LINDA'S
Azienda: LABORATORI OMEOPATICI LINDA'S S.R.L.
AIC: 046630
ABIES NIGRA
Azienda: BOIRON
AIC: 027142
ABILENE
Azienda: FARMACEUTICI FORMENTI S.P.A.
AIC: 036582
ABILIFY
Azienda: OTSUKA PHARMACEUTICAL NETHERLANDS BV
Clicked on next page
AIC: 043143
ABILIFY MAINTENA
Azienda: OTSUKA PHARMACEUTICAL NETHERLANDS BV
AIC: 025358
ABIMONO
Azienda: FARMITALIA INDUSTRIA CHIMICO FARMACEUTICA S.R.L.
AIC: 038637
ABINIX
Azienda: GENETIC S.P.A.
AIC: 024912
ABIOCEF
Azienda: NUOVO CONSORZIO SANITARIO NAZIONALE
AIC: 037350
ABIOCLAV
Azienda: AESCULAPIUS FARMACEUTICI SRL
AIC: 010772
ABIOSTIL
Azienda: LABORATORIO CHIMICO DECA DR.CAPUANI S.R.L.
AIC: 049479
ABIRATERONE ACCORD
Azienda: ACCORD HEALTHCARE, S.L.U.
AIC: 049583
ABIRATERONE KRKA
Azienda: KRKA D.D. NOVO MESTO
AIC: 049686
ABIRATERONE MYLAN
Azienda: MYLAN IRELAND LIMITED
AIC: 049298
ABIRATERONE TIEFENBACHER
Azienda: ALFRED E. TIEFENBACHER GMBH & CO. KG
AIC: 038038
ABIS
Azienda: A.G.I.P.S. FARMACEUTICI SRL
AIC: 021826
ABLACTON
Azienda: SCHERING SPA
AIC: 039399
ABRAXANE
Azienda: BRISTOL-MYERS SQUIBB PHARMA EEIG
AIC: 042292
ABRIFF
Azienda: MUNDIPHARMA PHARMACEUTICALS S.R.L.
AIC: 047231
ABROSYX N
Azienda: HOMEOSYN ITALIA S.R.L.
AIC: 046639
ABROTANUM
Azienda: BOIRON
Clicked on next page
AIC: 038494
ABSEAMED
Azienda: MEDICE ARZNEIMITTEL PUETTER GMBH & CO.KG
AIC: 036018
ABSORCOL
Azienda: ADDENDA PHARMA S.R.L.
AIC: 038736
ABSTRAL
Azienda: KYOWA KIRIN HOLDINGS B.V.
AIC: 033482
ACICLOVIR ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 025105
AMBROXOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035223
AMOXICILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036819
AMOXICILLINA E ACIDO CLAVULANICO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 033378
ATENOLOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040908
ATORVASTATINA ABC
Azienda: PHARMACARE S.R.L.
AIC: 038758
AZITROMICINA ABC
Azienda: IPSO PHARMA S.R.L.
AIC: 034350
BACAMPICILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039319
BROMAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035510
CAPTOPRIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036294
CARBOCISTEINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035361
CEFACLORO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035235
CEFONICID ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 035375
CEFOTAXIMA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035924
CEFTRIAXONE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037911
CETIRIZINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037929
CIPROFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036043
CITALOPRAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035129
CLODRONATO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036240
DIAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035809
DOMPERIDONE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 033876
DOXOFILLINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 038414
FINASTERIDE ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037931
FLUCONAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036519
GABAPENTIN ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036277
GENTAMICINA E BETAMETASONE ABC
Azienda: LABORATORIO FARMACOLOGICO MILANESE S.R.L.
AIC: 034584
GLUCONATO FERROSO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 044042
IBUPROFENE ABC FARMACEUTICI
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 038358
KETOROLAC ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 038064
LANSOPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039217
LATTULOSIO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040046
LEVOFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035931
LORAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039304
LORMETAZEPAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 042794
MEMANTINA ABDI
Azienda: ABDI FARMA UNIPESSOAL LDA
AIC: 037852
METAMIZOLO SODICO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039807
NEBIVOLOLO ABC
Azienda: PHARMACARE S.R.L.
AIC: 025000
NORFLOXACINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 040340
OLANZAPINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037665
OMEPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 041022
PANTOPRAZOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 025103
PIROXICAM ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 031837
PROPAFENONE ABBOTT
Azienda: ABBOTT S.R.L.
AIC: 038272
RAMIPRIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035505
RANITIDINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
Clicked on next page
AIC: 037278
RIFAXIMINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 043975
ROSUVASTATINA ABC
Azienda: IPSO PHARMA S.R.L.
AIC: 041742
SILDENAFIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037352
SIMVASTATINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 035134
SUCRALFATO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 045643
TADALAFIL ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 036950
TAMSULOSINA ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 037941
TRAMADOLO ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 018527
UREA ABBOTT
Azienda: ABBOTT S.R.L.
AIC: 040762
VALSARTAN ABC
Azienda: ABC FARMACEUTICI S.P.A.
AIC: 039486
VENLAFAXINA ABC
Azienda: BLUEFISH PHARMACEUTICALS AB
AIC: 026179
VERAPAMIL ABBOTT
Azienda: ABBOTT S.R.L.
Clicked on next page

Process finished with exit code 0