Rotating proxies don't get used in requests sent, but do rotate

Question

I'm trying to change proxies after x amount of requests for checking username availability against an api. However, my function for changing proxies doesn't seem to change the proxies in the requests I'm sending. In the proxy function the proxies do change. Can anyone figure out why the proxies don't work in the requests sent?

(old) Code:

proxy = ''
proxyFile = open(f'external/proxies.txt', 'r')
proxyList = [line.split(',') for line in proxyFile.readlines()]
...
    def proxies(self):
        try: 
            if self.proxyCount > 0:
                self.proxyCount += 1
            proxy = random.choice(self.proxyList)
            print('proxy in proxies function: ', proxy)
            return proxy
        except:
            pass

   def checkAccounts(self):
        while not self.usernames.empty():
            name = self.usernames.get(); self.usernames.put(name)
            url = f"https://public-ubiservices.ubi.com/v3/profiles?nameOnPlatform={name}&platformType=uplay"
            try:         
                r = requests.get(url, headers=self.headers, proxies=self.proxy)
                print('proxy in check function: ', self.proxy)
                
                try:
                    if self.checkedCount % 100 == 0:
                        self.proxies()
                except:
                    pass

                ctypes.windll.kernel32.SetConsoleTitleW(f"Gx | Checked: {self.checkedCount}, Available: {self.availableCount}, Errors: {self.errorCount}")
                if r.status_code == 200:
                    self.checkedCount += 1
                    if len(r.json()['profiles']) != 0:
                        print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Taken:          {name}")       
                    
                    else:  
                        print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Available:      {name}")
                        self.availableCount += 1

                if r.status_code == 429:
                    self.errorCount += 1
                    print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Error:          Rate limited")
                    self.proxies()
            
            except Exception:
                pass

Output:

[+] Taken:      name
proxy in proxies function:  ['222.74.202.234:80\n']
proxy in check function:

Previous post about the same topic

Edit1:

    with open('external/proxies.txt', 'r') as f:
        proxyList = [line.strip() for line in f]
...

    def proxies(self):
        if self.proxyCount > 0:
            self.proxyCount += 1
        proxy = random.choice(self.proxyList)
        print(proxy)
        return proxy

Output of Edit1:

85.214.65.246:80
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Gibbo\OneDrive\Bureaublad\[+] Projects\Main\[+] Ubi checker\ubichecker.py", line 66, in checkAccounts
    r = requests.get(url, headers=self.headers, proxies=self.proxy)
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 532, in request
    settings = self.merge_environment_settings(
  File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 710, in merge_environment_settings
    no_proxy = proxies.get('no_proxy') if proxies is not None else None
AttributeError: 'str' object has no attribute 'get'

abhijat · Accepted Answer · 2021-10-24T14:23:20.637

0

You are not assigning the return value of your proxy changing function to self.proxy. Fixing that should make the proxy change


proxyList = [proxyIp.strip() for line in proxyFile.readlines() for proxyIp in line.split(',') if proxyIp]

def proxies(self):
    try: 
        if self.proxyCount > 0:
            self.proxyCount += 1
        proxy = random.choice(self.proxyList)
        print('proxy in proxies function: ', proxy)
        return {
            'http': f'http://{proxy}',
            'https': f'http://{proxy}',
        }
    except:
        pass

def checkAccounts(self):
    # set the proxy initially, if not set already
    self.proxy = self.proxies()
    while not self.usernames.empty():
        name = self.usernames.get(); self.usernames.put(name)
        url = f"https://public-ubiservices.ubi.com/v3/profiles?nameOnPlatform={name}&platformType=uplay"
        try:         
            r = requests.get(url, headers=self.headers, proxies=self.proxy)
            print('proxy in check function: ', self.proxy)
            
            try:
                if self.checkedCount % 100 == 0:
                    self.proxy = self.proxies()
            except:
                pass

            ctypes.windll.kernel32.SetConsoleTitleW(f"Gx | Checked: {self.checkedCount}, Available: {self.availableCount}, Errors: {self.errorCount}")
            if r.status_code == 200:
                self.checkedCount += 1
                if len(r.json()['profiles']) != 0:
                    print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Taken:          {name}")       
                
                else:  
                    print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Available:      {name}")
                    self.availableCount += 1

            if r.status_code == 429:
                self.errorCount += 1
                print(f"{Fore.LIGHTBLACK_EX}[+]{Fore.RESET} Error:          Rate limited")
                self.proxy = self.proxies()
        
        except Exception:
            pass

EDIT: Seems your proxyList needs to be flattened first.

edited Oct 24 '21 at 14:23

answered Oct 24 '21 at 13:22

abhijat

535
6
12

Thanks, but the `proxies` function completely pauses the program. Can you figure why? @abhijat – Oct 24 '21 at 13:47
if I remove the `except: pass` I get this error: settings = self.merge_environment_settings( ```File "C:\Users\Gibbo\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 710, in merge_environment_settings no_proxy = proxies.get('no_proxy') if proxies is not None else None AttributeError: 'list' object has no attribute 'get'``` – Oct 24 '21 at 13:49
It looks like when you call `requests.get` the first time, the `proxies=` argument gets a list. And as per the `requests.get` code it expects a dict mapping protocols to proxies. Looking at your proxyList it is a list of lists, so it needs to be flattened. Can you post the content of your proxy file from where you parse out proxy IPs? – abhijat Oct 24 '21 at 14:12
output of your code: `proxy in proxies function: ['36.91.45.10:51672\n']` so it do it as a list – Oct 24 '21 at 14:15
I found a new way to get the proxies from the txt file, into a list, and then into a string. but it gives an error. (check the edit1 for info) – Oct 24 '21 at 14:21
Yes you need to package the proxy into a dict as per requests API as described here https://stackoverflow.com/a/35470245/1089912 - the latest edit packs the IP into a dict for http and https protoocols – abhijat Oct 24 '21 at 14:24
But I don't want to disable them, so why does it give that error in the edit1 and what's the solution for it – Oct 24 '21 at 14:28
1

The error is because you are passing a string to a function which expects a dict. You need to send a dict into `requests.get` as the proxies argument, which is what I edited my code to do. – abhijat Oct 24 '21 at 14:30

Rotating proxies don't get used in requests sent, but do rotate

1 Answers1