I am trying to scrape the links in a twitter post, but the link that I want is under different div tags. There are multiple parent div tags and I am only interested in the links under a specific parent div tag. The HTML I inspected shows me something like this:

Example of the HTML inspect I saw

I have searched for some answers here regarding conditionally finding elements by xpath for selenium. And the python code I am using to get the links that I want are:

[links.add(elem.get_attribute('href'))
for elem in browser.find_elements_by_xpath("//a[@dir ='ltr'] and //*[contains(@class, 'ABC')]")]

This the output I got "The result is not a node set, and therefore cannot be converted to the desired type. ":

selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression //a[@dir ='ltr'] and //*[contains(@class, 'r-1blvdjr')] because of the following error: TypeError: Failed to execute 'evaluate' on 'Document': The result is not a node set, and therefore cannot be converted to the desired type. (Session info: chrome=97.0.4692.71)

Could you please guide me on what I should do to rectify this to get my desired output which are the links under a specific parent div tag. This is my first time posting a question on stackoverflow, sorry if this question is lacking in anyway for people to understand what error I am facing currently.

CodePudding user response：

[links.add(elem.get_attribute('href'))
for elem in browser.find_elements_by_xpath("//*[contains(@class, 'ABC')]/descendant::a[@dir ='ltr']")]