0

I was making web-crawler crawling images from pixabay. At first, https 403 forbidden error occurred. so I added user agent in my code. but the same problem continued

this is my code to test error

from selenium import webdriver
import os
from selenium.webdriver.common.by import By
import urllib.request

current_path=os.getcwd()
driver=webdriver.Chrome(current_path+r'\chromedriver.exe')
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-logging"])
browser = webdriver.Chrome(options=options)
url=r'https://pixabay.com/ko/photos/search'

season=['spring', 'summer', 'autumm', 'winter']
new_url=url+'/'+season[0]+'%20풍경'
driver.get(new_url)

#useragent
header = {"my user agent"} 
req = urllib.request.Request(url, headers=header)
img_url=driver.find_element(By.CLASS_NAME,'photo-result-image').get_attribute('src')
workingdir_test='.\dataset\dataset_test'
urllib.request.urlretrieve(img_url, workingdir_test+season[0]+'.jpg')
Dongha.k.psy
  • 1
  • I don't even understand what you are trying to do... what is it your think that using a urllib.request.Request with a custom user agent does for the selenium webdriver? – Alexander Nov 23 '22 at 04:10
  • Does this answer your question? [Getting blocked by a website with selenium and chromedriver](https://stackoverflow.com/questions/66092682/getting-blocked-by-a-website-with-selenium-and-chromedriver) – Lucan Nov 23 '22 at 12:36

0 Answers0