Cannot scrape input from dynamic dropdown menu due to <span> (Python

this is a somewhat discussed problem, however I haven't find a solution in other posts.

I am creating an automation to submit parcels, using Selenium on Python, for this website: https://www.mondospedizioni.com

To select Country and City, I need to add an input on a text field. The text field is actually a search field, that opens a dropdown menu upon adding the key.

I fail to select the option and close the dropdown menu, thus filling the "Country"/"City" field and moving fwd.

The div I refer to (Country):

<span  id="select2-nazione_mittente-container" title="Selezionare Nazione di ritiro">Selezionare Nazione di ritiro</span>

The one for City:

<span >Inserire il CAP e selezionare la località</span>

This Post uses Select, but because of the tags I receive error:

Select only works on <select> elements, not on <span>.

I found another suggestion in this post from Anzel and even this one did not work for me.

My code gets stuck when I try to enter the sender's city, and the error is given to me when the code calls the input field for the receiver's country. My code:

import pandas as pd
import numpy as np
import time
from datetime import datetime, date

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

import itertools

driver = webdriver.Chrome(executable_path...)
url = "https://www.mondospedizioni.com/"
driver.get(url)
driver.set_page_load_timeout(-1) #Makes sure the webpage is fully loaded before scraping. Thus avoiding errors.

#All Page Components
hoCapito =driver.find_element(By.XPATH, '//*[@id="cookie-bar"]/p/a[1]')

countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-container"]')
cityFrom = driver.find_element(By.XPATH, '//*[@id="loc_mittente"]')
postFrom = driver.find_element(By.XPATH, '//*[@id="cap_mittente"]')

countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-container"]')
cityTo = driver.find_element(By.XPATH, '//*[@id="loc_destinatario"]')
postTo = driver.find_element(By.XPATH, '//*[@id="cap_destinatario"]')

myWeight = driver.find_element(By.XPATH, '//*[@id="peso1"]')
myLength = driver.find_element(By.XPATH, '//*[@id="l1"]')
myWidth = driver.find_element(By.XPATH, '//*[@id="p1"]')
myHeight = driver.find_element(By.XPATH, '//*[@id="h1"]')

calculate = driver.find_element(By.XPATH, '//*[@id="btn-calcola"]')


#MyKeys
driver = webdriver.Chrome(executable_path=r"C:\Users\AndreaPaviglianiti\Jupyter-2022\webdrivers\chromedriver.exe")
url = "https://www.mondospedizioni.com/"
driver.get(url)
driver.set_page_load_timeout(-1) #Makes sure the webpage is fully loaded before scraping. Thus avoiding errors.

hoCapito =driver.find_element(By.XPATH, '//*[@id="cookie-bar"]/p/a[1]')

countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-container"]')
cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-container"]/span')
postFrom = driver.find_element(By.XPATH, '//*[@id="cap_mittente"]')

countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-container"]')
cityTo = driver.find_element(By.XPATH, '//*[@id="loc_destinatario"]')
postTo = driver.find_element(By.XPATH, '//*[@id="cap_destinatario"]')

#Accept Cookies
hoCapito.click()


countryFrom.click()
select_countryFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Italia')
confirm_countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-results"]').click()

cityFrom.click()
select_cityFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Partinico')
confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div').click()


countryTo.click()
select_countryTo = driver.find_element(By.CLASS_NAME, 'select2-search_field').send_keys('Repubblica Slovacca')
confirm_countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-result-885w-Repubblica Slovacca"]').click()

cityTo.send_keys('Bratislava')
postFrom.send_keys(82101)

myWeight.send_keys(30)
myLength.send_keys(47)
myWidth.send_keys(25)
myHeight.send_keys(38)

calculate.click()
time.sleep(3)
driver.set_page_load_timeout(-1)

My error (now - based on my latest attempt):

<ipython-input-77-dd29b60fadf7>:2: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  driver = webdriver.Chrome(executable_path=r"C:\Users\AndreaPaviglianiti\Jupyter-2022\webdrivers\chromedriver.exe")

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
<ipython-input-77-dd29b60fadf7> in <module>
     25 cityFrom.click()
     26 select_cityFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Partinico')
---> 27 confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div').click()
     28 #confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div/text()').click()
     29 #confirm_cityFrom = driver.find_element(By.XPATH, '/html/body/span/span/span[2]').click()

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element(self, by, value)
   1242             value = '[name="%s"]' % value
   1243 
-> 1244         return self.execute(Command.FIND_ELEMENT, {
   1245             'using': by,
   1246             'value': value})['value']

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    422         response = self.command_executor.execute(driver_command, params)
    423         if response:
--> 424             self.error_handler.check_response(response)
    425             response['value'] = self._unwrap_value(
    426                 response.get('value', None))

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    245                 alert_text = value['alert'].get('text')
    246             raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 247         raise exception_class(message, screen, stacktrace)
    248 
    249     def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="select2-loc_cap_mittente-results"]/li/div"}
  (Session info: chrome=98.0.4758.81)
Stacktrace:
Backtrace:
    Ordinal0 [0x007F7AC3 2587331]
    Ordinal0 [0x0078ADD1 2141649]
    Ordinal0 [0x00683BB8 1063864]
    Ordinal0 [0x006B01CE 1245646]
    Ordinal0 [0x006B03CB 1246155]
    Ordinal0 [0x006DA612 1418770]
    Ordinal0 [0x006C86D4 1345236]
    Ordinal0 [0x006D8A0A 1411594]
    Ordinal0 [0x006C84A6 1344678]
    Ordinal0 [0x006A53F6 1201142]
    Ordinal0 [0x006A62E6 1204966]
    GetHandleVerifier [0x0099DF22 1680738]
    GetHandleVerifier [0x00A50DBC 2413564]
    GetHandleVerifier [0x0088D151 563089]
    GetHandleVerifier [0x0088BF13 558419]
    Ordinal0 [0x0079081E 2164766]
    Ordinal0 [0x00795508 2184456]
    Ordinal0 [0x00795650 2184784]
    Ordinal0 [0x0079F5BC 2225596]
    BaseThreadInitThunk [0x77CDFA29 25]
    RtlGetAppContainerNamedObjectPath [0x77E27A9E 286]
    RtlGetAppContainerNamedObjectPath [0x77E27A6E 238]

CodePudding user response：

#select2-nazione_mittente-container

Simply click the above element

.select2-search__field

Send keys to the above element

//li[@class='select2-results__option' and contains(@id,'Italia')]

Then proceed to click on the li tag with that value.

It's how to deal with any non Select dropdown element.