I'm trying to scrape a website using Selenium. I'm able to open most of the websites. But there are some websites that give me the response 406 Not Acceptable. Instead of receiving the actual website I only get a 406 Not Acceptable message in the browser.

I'm currently using the following Python code (using a Mac):

from selenium import webdriver
DRIVER_PATH = '/Users/User123/Downloads/chromedriver 2'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://www.WEBSITE.com/')

I tried to open the same website with the requests and beautiful soup libraries just out of curiosity and there it worked (I didn't receive that error). Nevertheless, I need selenium for my use case due to dynamic content on the page.

I googled it but didn't find any resources that cover the selenium vs. 406 issue.

Any ideas?

CodePudding user response：

They might be blocking Selenium? It might not be the case but try using this driver: https://github.com/ultrafunkamsterdam/undetected-chromedriver