Scraping all hrefs in concrete div types with Beautiful Soup-CodePudding

I am trying to create an app that would allow a person to get a list of GitHub repositories relevant to the search keywords they provide. On the result page for the search query the repositories have a special div class, namely:

<div >
      </div>

How to make the Beautiful Soup iterate through all of these classes on the page and then through all the <a> tags in search for hrefs?

For now I only have an idea how to get all the hrefs from <a>s:

import requests, sys, webbrowser, bs4

#variables
linkList = []


#handle input
print('Your GitHub repository search query:')
userInput = input()

#get the results from the URL

results = requests.get('https://github.com/search?q='   userInput   '&type=repositories'
                         ' '.join(sys.argv[1:]))
results.raise_for_status()

soup = bs4.BeautifulSoup(results.text, 'html.parser')

#find all the viable URLs

data = soup.find_all('a')

for aHref in data:
    if "href" in str(aHref):
        linkList.append(aHref)
        
        
print(linkList)

CodePudding user response：

This should get you pretty close.

import requests, sys, webbrowser, bs4

#variables
linkList = []


#handle input
print('Your GitHub repository search query:')
userInput = input()

#get the results from the URL

results = requests.get('https://github.com/search?q='   userInput   '&type=repositories'
                     ' '.join(sys.argv[1:]))
results.raise_for_status()

soup = bs4.BeautifulSoup(results.text, 'html.parser')

divs = soup.find_all('div', {'class': 'f4 text-normal'})
for div in divs:
    a_tags = div.find_all('a')
    for a_tag in a_tags:
        try:
            linkList.append(a_tag['href'])
        except:
            continue

# Test
for link in linkList:
    print(link)

CodePudding user response：

Note: Your selection is not that specific, it will also find other not expected links to.

Select your elements more specific and get its href attribute with a list comprehension - Access a tag’s attributes by treating it like a dictionary --> aHref['href]

['https://github.com/' a['href'] for a in soup.select('.repo-list-item .f4 a[href]')]

Example

import requests, sys, webbrowser, bs4

print('Your GitHub repository search query:')
userInput = input()

results = requests.get('https://github.com/search?q='   userInput   '&type=repositories'
                         ' '.join(sys.argv[1:]))
results.raise_for_status()

soup = bs4.BeautifulSoup(results.text, 'html.parser')

linkList = ['https://github.com/' a['href'] for a in soup.select('.repo-list-item .f4 a[href]')]

Output

['https://github.com//TheAlgorithms/Python',
 'https://github.com//geekcomputers/Python',
 'https://github.com//walter201230/Python',
 'https://github.com//injetlee/Python',
 'https://github.com//kubernetes-client/python',
 'https://github.com//Show-Me-the-Code/python',
 'https://github.com//xxg1413/python',
 'https://github.com//jakevdp/PythonDataScienceHandbook',
 'https://github.com//joeyajames/Python',
 'https://github.com//docker-library/python']