Selenium crawler automate next button for 100 pages using Python-CodePudding

I'm crawling a page for all the ID's (3000 ) which i'm later saving into a text file.

So my code is working the way i want it to, but now i'm searching for an automated method instead of repeating the below process 100 times, clicking through the webpage. Something like:

next.click()*100.....

Constructive advice is greatly appreciated :)

My current (working) code:

next = driver.find_element(By.CSS_SELECTOR,'img.down')
#next page button

main = driver.find_elements(By.CSS_SELECTOR, "span.id")
time.sleep(2)
for m in main:
    print(m.text)

next.click()

time.sleep(2)
main = driver.find_elements(By.CSS_SELECTOR, "span.id")

for m in main:
    print (m.text)

driver.quit()

CodePudding user response：

Thanks for the tip! A while loop using WebDriverWait does the trick.

wait = WebDriverWait(driver, 3)
while True:
        main = driver.find_elements(By.CSS_SELECTOR, "span.id")
        time.sleep(2)
        for m in main:
            print(m.text)
        try:
                element = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'img.down')))
                element.click()
        except TimeoutException:
                break

driver.quit()