Downloading images with selenium and requests: why does the .get_attribute() method of a WebElement-CodePudding

The

The program downloads the first couple of photos correctly, but then, suddenly, crashes.

Looking for solutions on stackoverflow, I have found this post: Requests : No connection adapters were found for, error in Python3

The answer of the post above suggests that the problem arises because of a newline charachter in the URL.

I checked the source URLs of the photos im the HTML code that couldn't be downloaded. They seem to be OK.

The problem seems to be either the browser.find_elements() method which parses the 'src' attribute values incorrectly or the .get_attribute() method which cannot fetch some of the URLs correctly. Instead of getting something like

https://c.tutti.ch/images/23452346536.jpg

the method gives back strings like

data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7

Of course, this is not a valid URL which the requests.get() method can use to download the image. I did some research and I have found out that this might be a base64 string...

Why does the .get_attribute() method return a base64 string in some of the cases? Can I prevent it to do so? Or do I have to convert it to a normal string?

CodePudding user response：

Can I suggest not using Selenium, there is a backend api that serves the data for each page. The only tricky thing is that requests to the api need to have a certain uuid hash which is in the HTML of the landing page. So you can get that when you go to the landing page, then use it to sign your subsequent api calls, here is an example which will loop through the pages and images for each post:

import requests import re import os search = 'Gartenstuhl' headers = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} url = f'https://www.tutti.ch/de/li/ganze-schweiz?q={search}' step = requests.get(url,headers=headers) print(step) uuids = re.findall( r'[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}',step.text) print(f'tutti hash code: {uuids[0]}') #used to sign requests to api os.makedirs('tuttiBilder', exist_ok=True) for page in range(1,10): api = f'https://www.tutti.ch/api/v10/list.json?aggregated={page}&limit=30&o=1&q={search}&with_all_regions=true' new_headers = { 'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36', 'x-tutti-hash':uuids[0], 'x-tutti-source':'web latest-staging' } resp = requests.get(api,headers=new_headers).json() for item in resp['items']: for image in item['image_names']: image_url = 'https://c.tutti.ch/images/' image pic = requests.get(image_url) with open(os.path.join('tuttiBilder', os.path.basename(image)),'wb') as f: f.write(pic.content) print(f'Saved: {image}')

CodePudding user response：

The program crashes, as you attempt to download a file (image) using a base64 encoded string (which is not a valid image URL). The reason these base64 strings show up in your images list is that each image (in <img> tags) appears to initially be a base64 string, and once it is loaded, the src value changes to a valid image url (you can check that by opening the DevTools in your browser while accessing your website at https://...ganze-schweiz?q=Gartenstuhl, and searching for "base64" in the Elements section of the DevTools. By moving to the next image in the search findings - using the arrow buttons - you'll notice the behaviour described above). This is also the reason (as shown in your cmd window snippet, and as tested it myself as well) that only 3 to 5 images are found and downloaded. That is because these 5 images are the ones appearing at the top of the page, and are succesfully loaded and given a valid image URL, when the page is accessed; whereas, the remaining <img> tags still include a base64 string.

So, the first step is - once the "search results" operation is completed- to slowly scroll down the page, in order for every image in the page to be loaded and given a valid URL. You can achieve that by using the method described here. You can adjust the speed as you wish, as long as it allows items/images to load properly.

The second step is to ensure that only valid URLs are passed to requests.get() method. Although every base64 string will be replaced by a valid URL due to the above fix, there might still be invalid image URLs in the list; in fact, there seems to be one (which is not related to the items) starting with https://bat.bing.com/action/0?t..... Thus, it is prudent to check that the requested URLs are valid image URLs, before attempting downloading them. You can do that by using str.endswith() method, looking for strings ending with specific suffixes (extensions), such as ".png" and ".jpg". If a string in the images list does end with any of the above extensions, you can then proceed downloading the image. Working example is given below (please note, the below will download the images appearing in the first page of search results. If you require downloading further image results, you can extend the program to navigate to the next page and then repeat the same steps as below).

import requests, os from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC def scroll_down_page(browser, speed=40): current_scroll_position, new_height= 0, 1 while current_scroll_position <= new_height: current_scroll_position = speed browser.execute_script("window.scrollTo(0, {});".format(current_scroll_position)) new_height = browser.execute_script("return document.body.scrollHeight") browser = webdriver.Firefox() # Opens Firefox webbrowser #browser = webdriver.Chrome() # Opens Chrome webbrowser browser.get('https://www.tutti.ch/') wait = WebDriverWait(browser, 10) wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#onetrust-accept-btn-handler"))).click() # accepts cookies terms wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "._1CFCt > input:nth-child(1)"))).send_keys('Gartenstuhl') # enters search key word wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[id*='1-val-searchLabel']"))).click() # clicks submit button os.makedirs('tuttiBilder', exist_ok=True) scroll_down_page(browser) # Slow scrolling down the page images = browser.find_elements(By.TAG_NAME, 'img') # stores every img element in a list suffixes = (".png", ".jpg") for im in images: imageURL = im.get_attribute('src') # get the URL of the image if imageURL.endswith(suffixes): print('Downloading image %s...' % (imageURL)) res = requests.get(imageURL) # downloads the image res.raise_for_status() imageFile = open(os.path.join('tuttiBilder', os.path.basename(imageURL)), 'wb') # creates an image file for chunk in res.iter_content(100000): # writes to the image file imageFile.write(chunk) imageFile.close() print('Done.') browser.quit()