Maximum number of threads is stuck on 861

Question

I'm doing a for loop and, because every iteration takes a bit of time, I thought to put the iteration inside a function and start it thanks to the threading module like this:

def myfunc(arg):
    """do something with arg, that takes a bit"""

for n in range(1, 1000000):
    Thread(target = myfunc, args = (n, )).start()

Yes, I need to do a lot of iteration, but when I start the program (that obviously is way more complicated than the example I put here) it raise the exception RuntimeError: can't start new thread. So i changed my code as below to check when my computer can't start new threads like this:

def myfunc(arg):
    """do something with arg, that takes a bit"""

for index, n in enumerate(range(1, 1000000)):
    try:
        Thread(target = myfunc, args = (n, )).start()
    except:
        print(index)
        exit()

Thanks to this I can understand how many iteration my for loop did and the output was, like the title said, 861 and yes, I tried to execute my code a lot of time to be sure: it's always 861. Now my question is: why exactly 861? There is a specific reason? And if this number have a reason, I can get it from a little python script (maybe a function) since I want this code to be executed on every possible computer? Sorry for the strange question.

P.S. I don't know if it's useful but I'll tell here: I have a 2700x processor, 8 cores and 16 threads.

Edit: as some of you required, here is the minimal reproducible example of my code below.

from threading import Thread
import socket

target = 'localhost'

def is_opened(port):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    if s.connect_ex((target, port)) == 0:
        return True

    return False


def portscanner(ports):
    opened = []

    def _support(port):
        print(port)
        if is_opened(port):
            opened.append(port)

    threads = []

    for port in ports:
        t = Thread(target = _support, args = (port, ))
        threads.append(t)

    for thread in threads:  # Here i tried with the enumerate
        thread.start()      # Here raise the RuntimeError

    for thread in threads:
        thread.join()

    return opened


if __name__ == "__main__":
    print(portscanner(range(49152, 65536)))

As you can see I'm trying to create a portscanner and if I apply the example I wrote up to my code I can see that as output I get 861. I don't want to utilize a limitated number of threads like 100 or 200 as someone said, I want instead to utilize the max power that a PC has to make this as fast as possible. Disclaimer: I know port scanning is illegal but I'm trying this only for learn and for personal exercise. I will never scan others network, server, service or similar.

Why do you need to start that many theads? Your CPU can only handle 16 anyway... As to why? Simple, available resources ran out. On linux there can be a limit defined that you can check with `cat /proc/sys/kernel/threads-max`. — fredrik, Mar 28 '20 at 22:09
No, you're wrong. My CPU can handle more than 16 threads. I read of this online and software threads and hardware threads are different. Thus, if you don't believe me, why i arrived at 800+ threads running at the same time? — Gianla, Mar 28 '20 at 22:23
Can you provide a [mcve]? As an aside, be careful about using a bare `except` like that, see https://stackoverflow.com/questions/54948548/what-is-wrong-with-using-a-bare-except. — AMC, Mar 28 '20 at 23:18
You can definitely have plenty of software threads but fredrik's point is fundamentally valid: you created a bunch of threads, each of which uses memory, and Python refused to make any more. Why 861? That has to do with how much memory is available in whatever environment you're on, or the Python interpreter limit. Mine didn't stop at 861. I can't think of too many tasks that require making 861 threads like this. Usually a few is plenty, especially in Python where each process is single-threaded anyway thanks to the GIL. Please explain what you're trying to do and add a [mcve]. — ggorlen, Mar 28 '20 at 23:48
First of all: I know I don't have to use ```except``` like that but It's only in that case, I know the error and only that kind of error can be raised. Second, I'll add a minimal reproducible example of what i'm trying to do later, it's a portscanner however, and i only want it to be as performing as possible scanning as many ports as possible. — Gianla, Mar 29 '20 at 08:48

Maximum number of threads is stuck on 861

0 Answers0