Try /Except inside for loop not behaving as expected

Question

CODE :

def ValidateProxy(LIST_PROXIES):
    '''
    Checks if scraped proxies allow HTTPS connection
    '''

    for proxy in LIST_PROXIES:

        print('using', proxy)

        host, port = str(proxy).split(":")

        try:
            resp = requests.get('https://amazon.com', 
                                proxies=dict(https=f'socks5://{host}:{port}'),
                                timeout=6)

        except ConnectionError:
            print(proxy, 'REMOVED')
            LIST_PROXIES.remove(proxy)


    print(len(LIST_PROXIES), 'PROXIES GATHERED')

    if len(LIST_PROXIES) != 0:
        return LIST_PROXIES
    else:
        return None

INPUT :

['46.4.96.137:1080', '138.197.157.32:1080', '138.68.240.218:1080'.....] #15 proxies

OUTPUT :

using 46.4.96.137:1080
46.4.96.137:1080 REMOVED
using 138.68.240.218:1080
138.68.240.218:1080 REMOVED
using 207.154.231.213:1080
207.154.231.213:1080 REMOVED
using 198.199.120.102:1080
198.199.120.102:1080 REMOVED
using 88.198.24.108:1080
88.198.24.108:1080 REMOVED
using 188.226.141.211:1080
188.226.141.211:1080 REMOVED
using 92.222.180.156:1080
92.222.180.156:1080 REMOVED
using 183.233.183.70:1081
183.233.183.70:1081 REMOVED
7 PROXIES GATHERED # len(LIST_PROXIES) == 7, so 8 are removed which are printed above

MY DOUBTS :

Why print('using', proxy) is not getting executed everytime ? (becuase input list has 15 items and this line is printed only 8 times)
Are try and except both blocks getting executed everytime ? Becuase everytime REMOVED is printed on console.
I want to function it like print('using', proxy) for every proxy and if ConnectionError then print(proxy, 'REMOVED') and remove that proxy from list.

EDIT : FULL INPUT

['46.4.96.137:1080', '138.197.157.32:1080', '138.68.240.218:1080', '162.243.108.129:1080', '207.154.231.213:1080', '176.9.119.170:1080', '198.199.120.102:1080', '176.9.75.42:1080', '88.198.24.108:1080', '188.226.141.61:1080', '188.226.141.211:1080', '125.124.185.167:38801', '92.222.180.156:1080', '188.166.83.17:1080', '183.233.183.70:1081']

https://www.programiz.com/python-programming/exception-handling — PM 77-1, Jun 09 '20 at 20:39
When you say "*not as expected*" it'a always helpful to show what your expectations are based on. — PM 77-1, Jun 09 '20 at 20:40
@PM77-1 expectations mentioned in point 3, if still unclear, please let me know, i will edit it — hack3r-0m, Jun 09 '20 at 20:42
The issue looks like in your except statement you remove the item from LIST_PROXIES, but you are iterating over LIST_PROXIES so when you remove the current proxy, let's say index 0, what was at index 1 is now index 0, but your next iteration grabs the new item at index 1. — L. MacKenzie, Jun 09 '20 at 20:46

Miguel Alorda · Accepted Answer · 2022-08-09T20:11:39.607

2

Edit 2022-08-09

I would separate the logic into two functions. Also, please follow PEP-8 (I did not point that in the original answer)

from typing import Iterable

import requests

def is_valid_proxy(proxy: str) -> bool:
    try:
        requests.get(
            'https://amazon.com',
             proxies={'https': f'socks5://{proxy}'},
             timeout=6,
        )
        return True
    except ConnectionError:
        return False


def get_valid_proxies(proxies: Iterable[str]) -> list[str]:
    return [proxy for proxy in proxies if is_valid_proxy(proxy)]

Instead of printing to stdout, you could use the logging module.

Original Answer

The problem is you are iterating over the LIST_PROXIES and removing elements from it at the same time.

If you only want to iterate over the LIST_PROXIES once, something like this could work:

def ValidateProxy(LIST_PROXIES):
    index = 0 
    for i in range(len(LIST_PROXIES)):
        proxy = LIST_PROXIES[index]
        print('using', proxy)
        host, port = str(proxy).split(":")
        try:
            resp = requests.get('https://amazon.com', 
                                proxies=dict(https=f'socks5://{host}:{port}'),
                                timeout=6)
            index += 1
        except ConnectionError:
            print(proxy, 'REMOVED')
            LIST_PROXIES.pop(index) # Index is not incremented
    print(len(LIST_PROXIES), 'PROXIES GATHERED')
    if len(LIST_PROXIES) != 0:
        return LIST_PROXIES
    else:
        return None

However, if iterating over the list twice is not a problem, you can just make a copy of the list, as Sy Ker pointed out.

edited Aug 09 '22 at 20:11

answered Jun 09 '20 at 20:53

Miguel Alorda

623
5
13

Thanks for this, i understood the approach but gives index out of range exception when replacing ```proxy``` with ```LIST_PROXIES[i]``` (replacing becuase proxy will be undefined) . however it can be tweaked :). And yes, creating duplicate list is very costly in this case as sizes may be very large. – hack3r-0m Jun 09 '20 at 21:05
Thank you for pointing it out. If should be `LIST_PROXIES[index]`, let met edit the post! :) – Miguel Alorda Jun 09 '20 at 21:07
I'm not sure how my method is iterating twice over the data. – Sy Ker Jun 09 '20 at 21:09
1

@SyKer Because, to create a copy of a list, you first have to iterate over it. Then, after the copy has been created, you have to iterate over the elements to check whether they should be eliminated or not. – Miguel Alorda Jun 09 '20 at 21:14
seems like there is one more bug with try/except, after fixing proxy removal, all items are iterated but both try and excpet block are executed, edited question with full input – hack3r-0m Jun 09 '20 at 21:19
How about 'for i, value in enumerate(list)' ? – Sy Ker Jun 09 '20 at 21:27
@hack3r_0m What does it mean that the both the try and except blocks are executed? The `requests.get` call will always be executed. If a `ConectionError` ocurrs, the execution will continue from the except block. However, if there is no exception, the `index` will be incremented by 1, and the except block will be skipped. – Miguel Alorda Jun 09 '20 at 21:29
@SyKer In the [official documentation](https://docs.python.org/3/library/functions.html#enumerate), it reads the list will be iterated. In general, every function you apply to an iterable will iterate over its elements at least once. – Miguel Alorda Jun 09 '20 at 21:33

score 1 · Answer 2 · answered Jun 09 '20 at 20:48

The issue is caused by the fact that you are mutating the list whilst you are still looping over it in this line.

LIST_PROXIES.remove(proxy)

This means that just before the for loop looks for the 'next' item in the list, the 'next' item moves left in the list and therefore is missed completely.

Check out this question/answer: strange result when removing item from a list

score 1 · Answer 3 · answered Jun 09 '20 at 20:49

1

You are removing items from a list you are iterating over. NOT GOOD. You should iterate over a copy of the list, leaving you free to modify the original. Simply replace for proxy in LIST_PROXIES: with for proxy in list(LIST_PROXIES):

answered Jun 09 '20 at 20:49

Sy Ker

2,047
1
4
20

Try /Except inside for loop not behaving as expected

3 Answers3

Edit 2022-08-09

Original Answer