Pagination with selenium and python-CodePudding

I'm trying to scrape using selenium and python, the web page have a paginator in javascript, when I click in the button, I can see that the content reload but when I try to get the new table information it's the same old table info, selenium doesn't noticed that de DOM info has changed, I'm aware about the stale DOM, I'm just looking for the best path to solve this problem

for link in source.find_all('div', {'class': 'company-row d-flex'}):
    print(link.a.text, link.small.text, link.find('div', {'class': 'col-2'}).text)

# Next button (I´ll make an iterator)
driver.find_element_by_xpath('//a[@href="hrefcurrentpage=2"]').click()

# Tried this and doesn't work
# time.sleep(5)

# Here the table change but get the same old info 
for link in source.find_all('div', {'class': 'company-row d-flex'}):
    print(link.a.text, link.small.text, link.find('div', {'class': 'col-2'}).text)  ```

CodePudding user response：

I think you are getting the same data after opening the next page even after delay since you are getting your data from the existing source.
So, you should re-read, reload the source after clicking the pagination, possibly with some delay.