Time out exception with Selenium WebDriverWait-CodePudding

all,

I'm pretty new to Python and Selenium so, I hope you are forgiving! I know this has been asked before. I'm trying to scrape data from this page but not being successful. I get a timeout error even though my wait time is pretty large. Here is my modest code:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


driver_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(executable_path=driver_path)
driver.header_overrides = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}
url = "https://superrare.com/features/series-4" 
response = driver.get(url)
html = driver.page_source
element = WebDriverWait(driver,50).until(EC.presence_of_element_located((By.CLASS_NAME, "FeaturedArtistName-sc-jssh89-2 iTyMhR")))

print(html)
driver.quit()

CodePudding user response：

Couple of issues here:

You are using a classname that has a lot of randomization in it. Chances are that it may change. So I tried to make it a little robust by taking only the static part of the classname.
There are 13 elements with the classname FeaturedArtistName-sc-jssh89-2 iTyMhR ( and I am presuming that you are trying to wait for all of them), but you are asking the driver to wait for the presence_of_element whereas it should be presence_of_elements. So, I tweaked your code a bit, and in a jiffy, I got the result.

This is the line I tweaked: element = WebDriverWait(driver,50).until(EC.presence_of_all_elements_located((By.XPATH, "//*[contains(@class,'FeaturedArtistName')]")))

Output (I am not showing all the output here to save space)

<html lang="en" ><head><script type="text/javascript" async="" src="https://widget.intercom.io/widget/d3gh4rmg"></script><script type="text/javascript" async="" src="https://www.google-analytics.com/analytics.js"></script><script id="twitter-wjs" src="https://platform.twitter.com/widgets.js"></script><script>!function(){var t=window,e=t.Intercom;if("function"==typeof e)e("reattach_activator"),e("update",t.intercomSettings);else{var