Python selenium - no search results showing when search button is clicked-CodePudding

I have successfully automated Edge (and Chrome) to the correct page for searching broadband prices and also entered a postcode into the search box. However, when I click the search button, it doesn't show me the results of that search. I need to get the results for several postcodes and scrape the results to pandas eventually. I have identified the search button successfully (I believe) via different methods (Xpath, CSS selector) but the clicking and returning of results is not working automatically. The website works fine when used normally (not automating it).

Here is my python code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

#Other imports here
import os
import wget


driver = webdriver.Edge('msedgedriver.exe')

driver.get("https://www.thinkbroadband.com/packages")

# Cookie consent button click
consent = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[3]/div[2]/div[1]/div[2]/div[2]/button[2]"))).click()
consent2 = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[3]/div[2]/div[2]/div[3]/div[2]/button[1]"))).click()

# inputting postcode
postcode_input_box = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[2]/div[7]/div/div/div/form[1]/div[1]/input")))
postcode_input_box.send_keys("se19qu")

# identify search button
search_button = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='broadband-package-search']/form[1]/div[2]/input")))

# Click the search button - neither method works
#search_button.send_keys(Keys.ENTER)
search_button.click()

I know there's some process of interrogating a database of results for the postcode but I'm not sure how to automate that without using the web page elements I've used here.

Many thanks.

CodePudding user response：

When I tried your code, the cookie elements were not showing up, and hence the cookie consent elements were failing. I used a try/except to overcome this. It works:

driver.get("https://www.thinkbroadband.com/packages")
# Cookie consent button click
try:
    consent = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[3]/div[2]/div[1]/div[2]/div[2]/button[2]"))).click()
    consent2 = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[3]/div[2]/div[2]/div[3]/div[2]/button[1]"))).click()
except:
    print("Cookie Consent Elements Not Found")
    pass
# inputting postcode
postcode_input_box = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[2]/div[7]/div/div/div/form[1]/div[1]/input")))
postcode_input_box.send_keys("se19qu")
# identify search button
search_button = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='broadband-package-search']/form[1]/div[2]/input")))
# Click the search button - neither method works
#search_button.send_keys(Keys.ENTER)
search_button.click()

# trying to load into a dataframe (a rough task here. Please refine per your requirement
eles = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='packages']//tr")))
ls = [ele.text for ele in eles]
# print(ls)
df = pd.DataFrame(ls)
print(df)

Output:

Cookie Consent Elements Not Found
                                                     0
0    Provider\nPackage\nDownload Speed\n(average)\n...
1    WOW!\nby Zzoomm\nSpecial package\n2,000\nMbps\...
2    Virgin Media Gig1 Fibre Broadband\nby Virgin M...
3    Virgin Media Gig1 Fibre Broadband   Phone\nby ...
4    Ultrafast Ultimate\nby IDNet\nSpecial package\...
..                                                 ...
437  Unlimited Business Broadband **1 month contrac...
438  Unlimited Business Broadband   Phone Package *...
439  Unlimited Business Broadband Office   Phone Pa...
440  Pure Fibre Lite\nby PureFibre (Also Derwenthor...
441  Vodafone Unlimited Lite 5G\nby Vodafone Broadb...

[442 rows x 1 columns]

Process finished with exit code 0

CodePudding user response：

Maybe try using the class or the css selector in my experience the css selector works when the xpath doesn't

CSS selector: .submit