0

So for example, I'm making an async decorator and wanted to limit the number of concurrent threads:

from multiprocessing import cpu_count
from threading import Thread

class async:
    def __init__(self, function):
        self.func = function
        self.max_threads = cpu_count()
        self.current_threads = []
    def __call__(self, *args, **kwargs):
        func_thread = Thread(target = self.func, args = args, kwargs = kwargs)
        func_thread.start()
        self.current_threads.append(func_thread)
        while len(self.current_threads) > self.max_threads:
            self.current_threads = [t for t in self.current_threads if t.isAlive()]


from time import sleep

@async
def printA():
    sleep(1)
    print "A"

@async
def printB():
    sleep(1)
    print "B"

Is this going to limit the total concurrent threads? IE. If I had 8 cores, would the current code end up having 16+ threads due to two separate async objects existing?

If so, how would I fix that? Thanks!

martineau
  • 119,623
  • 25
  • 170
  • 301
Ben
  • 159
  • 8
  • For one thing, threads don't use multiple cores, so relating what you're trying to do to the number of them doesn't make sense. – martineau Jan 24 '17 at 23:21
  • Oh. Coming from a Java background, that was the way it worked there so I assumed it was the same. Then I have two questions: 1. What is the proper way to determine an appropriate max_concurrent_threads? 2. I looked around and saw the proper way to take advantage of multiple cores is this: http://stackoverflow.com/questions/23537037/python-multicore-programming In which case, does that make use of threads, and if not, would there be any benefit in doing so? – Ben Jan 25 '17 at 00:03
  • The only way to do multi-core processing in Python is to use the multiprocessing module. Threading mostly gives an illusion of concurrent processing since all threads share the same interpreter running on a single core - the GIL or Global Interpreter Lock enforces this policy. For the most part, real concurrency only happens when one of the threads does some I/0. The biggest positive thing it has going for it (IMO) is that the sharing of data is easier if done properly and therefore it's a little easier to use. The optimal maximum number of threads would depend on exactly what was being done. – martineau Jan 25 '17 at 03:29

0 Answers0