0

I am trying to use proxies to scrape some informations off a website, however I have heard that urlib.getProxies() doesn't work, especially for https websites.Here is my code: 

page = requests.get(url,proxies = urllib.getproxies())

So does this actually work and is there a way to find out if it is requesting through proxies, instead of my ip address.By the way I am requesting a https website. Thanks for your help ahead of time.

Nazariy
  • 717
  • 6
  • 23

1 Answers1

1

this question & answer explains exactly what your problem might be.

The proxyDict you're supplying to requests.get(... proxies = X) needs to have a proxy set for the specific protocol of the request you're making. That is, if you're making an https requests, you need to have proxyDict["https"] set to something.

What does urllib.getproxies() return for you? Is there anything set for the https protocol?

Community
  • 1
  • 1
will
  • 10,260
  • 6
  • 46
  • 69