Python3 Requests Proxy (Socks5)

Question

Edit: I don't know if this is a thing, but often this happens when i've been testing a lot and I have a lot of python scripts running. Usually none active at the same time but I'll have ran a few of them in a short space of time. I don't know if that's why teh error starts happening.

Ok here's my dilemma. I've got a NordVPN account and I want to randomly loop through the ip's for the requests i'm making to google. This works fine .. sometimes. For some reason after a few request pings my program starts slowing down and then I start getting 'Max Connection Exceeded' errors and as far as I can tell the ip's are fine.

I started thinking that google was blocking the ip but then I request to other website and the same error continues.. I started thinking that maybe the proxy server only allows a certain amount of requests in a certain time frame so I tried some free proxies but I get the same errors again.. Either way if I leave it alone for a while it seems to work fine and then a few dozen pings later and the errors start again.

The only think I can think of and not sure how I can test or even if it's possible to test remotely (work router) is that maybe some open connections remain open and affects my code and then when the backlog of open connections are thrown out I can resume as normal. I started looking for if I needed to "close" my connection after my get request but it doesn't seem necessary (although I wasn't even sure what I was looking for).

This is one version of my code, I've tried without sessions, I've tried a different way of writing the sessions. All seem to work in the same way:

import requests, random

username = 'xxx'
password = 'yyy'

with open('proxies.txt', 'r') as p:
    proxy_lost = p.read().splitlines()
with open('data.txt', 'r') as d:
    data = d.read().splitlines()

for i in data:
    result = None
    while result is None:
        try:
            proxy = proxy_list[random.randint(0,len(proxy_list)-1)] + '.nordvpn.com'
            prox = 'socks5://{}:{}@{}:{}'.format(username, password, proxy, 1080) #I've also tried socks5h

            with requests.Session() as r:
                r.proxies['http': prox]
                r.proxies['https': prox]

                result = r.get('http://icanhazip.com')
                print(result.text.strip())

If anyone has any idea whatsoever any help is appreciated. I've reached a point where i'm struggling to think of new ideas to try.

This is an example of one of the errors I get during this whole process:

Traceback (most recent call last): File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 809, in connect negotiate(self, dest_addr, dest_port) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 444, in _negotiate_SOCKS5 self, CONNECT, dest_addr) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 503, in _SOCKS5_request raise SOCKS5AuthError("SOCKS5 authentication failed") socks.SOCKS5AuthError: SOCKS5 authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py", line 88, in _new_conn **extra_kw File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 209, in create_connection raise err File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 199, in create_connection sock.connect((remote_host, remote_port)) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 47, in wrapper return function(*args, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py", line 814, in connect raise GeneralProxyError("Socket error", error) socks.GeneralProxyError: Socket error: SOCKS5 authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen chunked=chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request conn.request(method, url, **httplib_request_kw) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output self.send(msg) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send self.connect() File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 181, in connect conn = self._new_conn() File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py", line 110, in _new_conn "Failed to establish a new connection: %s" % error urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: SOCKS5 authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: SOCKSHTTPConnectionPool(host='icanhazip.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: SOCKS5 authentication failed'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users*\Documents_Scripts\Find Proxies\FindProxies.py", line 23, in result = requests.get('http://icanhazip.com', proxies=proxies) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 75, in get return request('get', url, params=params, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: SOCKSHTTPConnectionPool(host='icanhazip.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: SOCKS5 authentication failed'))

Is this actually still working for you at all Naiek with NordVPN and using SOCKS via requests? — Timothy Dalton, Aug 20 '20 at 11:09
It doesn't. I gave up on NordVPN for scraping. I'm guessing that because it's meant to be used as a VPN only, they limit the amount of connection requests you can make. And considering the nature of requests, it stopped it from working after a while. When and if I get back into the project I'll have to buy a dedicated IP to make it work properly. — Naiek, Sep 07 '20 at 15:40
Thanks and I can confirm, I couldn't get this working either. — Timothy Dalton, Sep 07 '20 at 16:10
@Naiek hey man, how did you get the nordvpn proxy list? their website only shows the top servers and not the full list :( ( the server needs to be socks5 enabled so many of their servers don't work as a proxy) — OneAndOnly, Sep 21 '20 at 08:35
@OneAndOnly Hi. The link is: https://nordvpn.com/servers/tools/ . Let's you filter by Country and then in Advanced options you can select the Protocol there. Hope this helps! — Naiek, Sep 22 '20 at 09:06
It will work if you don't randomly pick a different server for each iteration. They seem to be limiting the number of new connections to a new proxy server and not the number of requests. — Hans Daigle, Mar 26 '22 at 03:55

score 3 · Answer 1 · edited Nov 19 '19 at 10:04

3

Are you also closing the sessions again after use?

Try with:

r = requests.session(config={'keep_alive': False})

Inspired by: python-requests-close-http-connection

edited Nov 19 '19 at 10:04

tobsob

602
9
22

answered Nov 19 '19 at 09:31

Sempai

63
9

3

I believe I've tried that. The only conclusion I could come to is that NordVPN doesn't just stop a certain amount of connections being opened at once (hense the close session thought process), but that it actively limits my account to a certain amount of requests to a new proxy server in a certain time frame. Allowing an average user to connect to a new server no problem but stopping scrapers. As a side note I've since discovered that NordVPN does forbid their service to scapers. Buried deep in the T&C – Naiek Dec 04 '19 at 10:41

Python3 Requests Proxy (Socks5)

1 Answers1