Having Trouble Clicking in Date Field with Selenium-CodePudding

I'm trying to scrape a table from the 1/30/2022 slate. However, I get the 'unable to locate element' error when I attempt to click in the date field and change the date from 2/6 to 1/30. I've tried finding by class name as well. Is there another way to do this, or is there something I'm doing wrong?

from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
import time

path = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(path)

driver.get('https://rotogrinders.com/resultsdb/nfl')

time.sleep(5)
driver.maximize_window()

time.sleep(10)

search = driver.find_element_by_xpath('//*[@id="navbar-demo1-mobile"]/div[1]/div/span/div')
search.click()


previous = driver.find_element_by_class_name('react-datepicker__navigation react-datepicker__navigation--previous')
previous.click()

time.sleep(5)

date = driver.find_element_by_class_name('react-datepicker__day react-datepicker__day--030

react-datepicker__day--weekend') date.click()

CodePudding user response：

You are not able to find it because it is inside an iframe. You have to switch to iframe window first, and then try to access the element.

Also, I see that the date picker has enabled to key in the dates, so you could use send_keys to type in the date. It makes your code a little easier on you perhaps. But you may write to click on the date picker ui. It's your choice per se. Having said that here is the code:

driver.get("https://rotogrinders.com/resultsdb/nfl")
time.sleep(10)
frame = driver.find_element(By.XPATH, "//iframe")
driver.switch_to.frame(frame)
date_picker = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='react-datepicker__input-container']//input")))
date_picker.send_keys("01/16/2022")
time.sleep(10)

Try to change the time.sleep to explicit wait if possible (webdriverwait)

CodePudding user response：

wait=WebDriverWait(driver,60)
driver.get('https://rotogrinders.com/resultsdb/nfl')
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe")))
date = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.react-datepicker__input-container input')))
date.send_keys("01/16/2022")

First wait for the iframe and then proceed to click the search element and then send keys.

Import:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

CodePudding user response：

It might be possible to avoid Selenium here. It's just a matter of pulling out some id's to feed into the direct url.

import requests
import datetime
import pandas as pd

dateStr = input('Enter date (YYYY-MM-DD): ')
dateStr_alpha = datetime.datetime.strptime(dateStr, '%Y-%M-%d').strftime('%Y%M%d')

url = f'https://service.fantasylabs.com/contest-sources/?sport_id=1&date={dateStr}'
jsonData = requests.get(url).json()
groupId = jsonData['contest-sources'][0]['draft_groups'][0]['id']


url = f'https://service.fantasylabs.com/live-contests/?sport=NFL&contest_group_id={groupId}'
jsonData = requests.get(url).json()
contestId = jsonData['live_contests'][0]['contest_id']

url = f'https://dh5nxc6yx3kwy.cloudfront.net/contests/nfl/{dateStr_alpha}/{contestId}/data/'
jsonData = requests.get(url).json()


contestUsers = pd.DataFrame(jsonData['users']).T.reset_index(drop=True)
fieldExposures = pd.DataFrame(jsonData['players']).T


for k, v in jsonData['exposures'].items():
    exposureDf = pd.DataFrame(v['exposureCounts']).T
    exposureDf.columns = [x   f'_top_{k}%' for x in exposureDf.columns]
    fieldExposures = pd.merge(fieldExposures, exposureDf, how='left', left_index=True, right_index=True )
    
fieldExposures = fieldExposures.fillna(0).reset_index(drop=True)

Output:

print(contestUsers)
           userId  ...                                          exposures
0         ilanwor  ...  {'6957:0': {'playerId': '6957:0', 'playerCt': ...
1          IamPGM  ...  {'6957:0': {'playerId': '6957:0', 'playerCt': ...
2      lazgetsfit  ...  {'6957:0': {'playerId': '6957:0', 'playerCt': ...
3        kroug005  ...  {'6957:0': {'playerId': '6957:0', 'playerCt': ...
4          TwoGun  ...  {'6957:0': {'playerId': '6957:0', 'playerCt': ...
..            ...  ...                                                ...
153      Jquave76  ...  {'64500:0': {'playerId': '64500:0', 'playerCt'...
154  roxygirl9847  ...  {'33623:0': {'playerId': '33623:0', 'playerCt'...
155      Eddieron  ...  {'64500:0': {'playerId': '64500:0', 'playerCt'...
156     Grendal22  ...  {'64500:0': {'playerId': '64500:0', 'playerCt'...
157     afountain  ...  {'33623:0': {'playerId': '33623:0', 'playerCt'...

[158 rows x 17 columns]

print(fieldExposures)
    playerId firstName  ... exposureCt_top_1% exposurePerc_top_1%
0       1343     Odell  ...               3.0               100.0
1       1386    Travis  ...               0.0                 0.0
2       3134     Jimmy  ...               0.0                 0.0
3       3294      Josh  ...               0.0                 0.0
4       3573     Clark  ...               0.0                 0.0
..       ...       ...  ...               ...                 ...
93     76717      Sony  ...               0.0                 0.0
94     81679      John  ...               0.0                 0.0
95     88149      Jody  ...               0.0                 0.0
96    163000     Chris  ...               0.0                 0.0
97    181636      Mark  ...               0.0                 0.0

[98 rows x 26 columns]