Python: HTTPS using a proxy server with requests

Question

I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.

I can access https websites fine with a browser but I cannot use requests to get the pages

This works:

import requests
IP=requests.get('http://api.ipify.org').text

proxyDict = { "http"  : IP,
              "https"  : IP
            }

url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)

This doesn't:

url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)

only difference is http vs https

here is the traceback:

File "C:\Python27\lib\site-packages\requests\adapters.py", line 502, in send
    raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))

I've tried other https websites, they all don't work.

I have also emailed support at MyIPHide. They said all proxies support https, which is true when I use a browser only.

One work around that works is if I use Selenium and get the page, then use driver.page_source for text.

It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.

I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:

import requests
url='https://www.cnn.com'
r=requests.get(url)

>>> r
<Response [200]>
>>>

Also tried python 3.6 with requests 2.20.1 --> same results.

did you add the port? {"https":"ip:port"}. Or try `{"https":"183.88.219.163:57457"}` — KC., Nov 15 '18 at 04:11
If my proxy work for you that means the issue occured on your proxy — KC., Nov 15 '18 at 04:15
@kcorlidy. Thanks for the proxy. It looks like it also doesn't work. `ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError(': Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))` — jason, Nov 15 '18 at 14:24
yes, not all nodes work for me too. But also you can find one on https://free-proxy-list.net/. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution. — KC., Nov 15 '18 at 14:35
@kcorlidy. I think the problem is here. Python2 problem. https://stackoverflow.com/questions/40247025/flask-socket-error-errno-10053-an-established-connection-was-aborted-by-the . it seems like the `app.run(threaded=True)` would be the easist. How do I put that in the code? — jason, Nov 15 '18 at 14:54
@jason, how did you solve this? I am having same issue. Requests 2.22 — workin 4weekend, Jul 29 '19 at 20:25

score 2 · Answer 1 · answered Nov 15 '18 at 15:35

2

On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter(). The way like that(to ensure if it works you can set to 0).

import requests
from requests.adapters import HTTPAdapter

proxy = {"http":"118.174.233.31:51726",
        "https":"43.254.132.86:50659"} # free proxy,often need to modify

with requests.Session() as se:
    # pool_connections=pool_maxsize=0 -> pool closed
    se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100)) 
    se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
    print(se.adapters["https://"]._pool_maxsize)
    #print(se.get("https://github.com"))
    print(se.get('http://api.ipify.org',proxies=proxy).text)

#100
#118.174.233.31

answered Nov 15 '18 at 15:35

KC.

2,981
2
12
22

Ran on Windows10, Python27, requests-2.20.1 – KC. Nov 15 '18 at 15:41
1

Thanks for trying, but I still get the same error when I make a page request. `url='https://www.cnn.com' r=se.get(url,proxies=proxy)` gives me `ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))` – jason Nov 15 '18 at 16:02
what about your requests_version, and can you access website without https proxy? – KC. Nov 15 '18 at 16:16
yes I can. `requests == 2.18.4`. I just upgraded to `2.20.1`. Same error. I'm running python 2.7.15 – jason Nov 15 '18 at 16:32

Python: HTTPS using a proxy server with requests

1 Answers1