How to handle proxies in urllib3

Question

I am having trouble finding solid examples of how to build a simple script in urllib3 which opens a url (via a proxy), then reads it and finally prints it. The proxy requires a user/pass to authenticate however it's not clear to me how you do this? Any help would be appreciated.

I don't know the library, but is it reasonably similar to [Proxy with urllib2](http://stackoverflow.com/q/1450132) ? — Nathan Tuggy, Jul 01 '15 at 03:12
Not really, I have done this will the urllib2 library already. I am interested to see how this is done with urllib3. — Tom, Jul 01 '15 at 03:21
use `requests`? http://stackoverflow.com/questions/8287628/proxies-with-python-requests-module — maxymoo, Jul 01 '15 at 07:15

score 16 · Accepted Answer · edited Dec 28 '20 at 18:02

16

urllib3 has a ProxyManager component which you can use. You'll need to build headers for the Basic Auth component, you can either do that manually or use the make_headers helper in urllib3.

All together, it would look something like this:

from urllib3 import ProxyManager, make_headers

default_headers = make_headers(proxy_basic_auth='myusername:mypassword')
http = ProxyManager("https://myproxy.com:8080/", proxy_headers=default_headers)

# Now you can use `http` as you would a normal PoolManager
r = http.request('GET', 'https://stackoverflow.com/')

edited Dec 28 '20 at 18:02

tyleha

3,319
1
17
25

answered Jul 01 '15 at 09:08

shazow

17,147
1
34
35

Is the "headers" the login for the proxy, or can it be used to log into a website as well? – SPYBUG96 Dec 05 '17 at 16:12
1

@SPYBUG96 It's for the proxy. You can make separate headers for the website itself. – shazow Dec 05 '17 at 20:11
This only worked for me when I changed "headers=default_headers" to "proxy_headers=default_headers". See https://stackoverflow.com/questions/58131573/urllib3-proxymanager-is-giving-authentication-error – Kyle Apr 03 '20 at 01:54

score 6 · Answer 2 · edited May 03 '19 at 20:23

I believe the correct answer to this should be

from urllib3 import ProxyManager, make_headers

default_headers = make_headers(proxy_basic_auth='myusername:mypassword')
http = ProxyManager("https://myproxy.com:8080/", headers=default_headers)

# Now you can use `http` as you would a normal PoolManager
r = http.request('GET', 'https://stackoverflow.com/')

(note: proxy_basic_auth, not basic_auth)

I was trying this with basic_auth in my environment without any luck. shazow you committed this comment to git which pointed me in the right direction

How to handle proxies in urllib3

2 Answers2

Linked

Related