0

I have a method defined inside a class as follows:

from Queue import Queue
from threading import Thread, Lock

class Example(object):
    def f(self):
        things = [] # list of some objects to process

        my_set = set()
        my_dict = {}
        my_list = []

        def update(q, t_lock, my_set, my_dict, my_list):
            while True:
                thing = q.get()

                new_set, new_dict, new_list = otherclass.method(thing)

                with t_lock:
                    my_set = set_overlay(my_set, new_set)
                    my_dict.update(new_dict)
                    my_list += new_list

                q.task_done()

        q = Queue()
        num_threads = 10
        thread_lock = Lock()

        for i in range(num_threads):
            worker = Thread(target=update, args=(q, thread_lock, my_set, my_dict, my_list))
            worker.setDaemon(True)
            worker.start()

        for t in things:
            q.put(t)

        q.join()

As you can see I'm trying to update the variables my_set, my_dict, and my_list with some results from the threaded update method defined inside the f method. I can pass them all to the update function, but this only works for mutable datatypes.

I updated this to use threading.Lock() as user maxywb suggested, but I would like to keep the same questions below open to be answered now that the lock is included.

My questions are:

  1. Is this threadsafe? Can I guarantee that no updates to any of variables will be lost?
  2. What if I wanted to throw an immutable variable into the mix like an int, that was added to from the results of otherclass.method(thing)? How would I go about doing that?
  3. Is there a better way to do this/architect this? The idea here is to update variables local to a class method from a thread, while sharing a (hopefully) threadsafe reference to that variable across threads.
Nat Dempkowski
  • 2,331
  • 1
  • 19
  • 36
  • Try looking here:http://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe. You need one or more mutexes to control access to the variables. – maxywb Jun 02 '14 at 21:38
  • @maxywb So I updated it to use `threading.Lock()`, any chance you could take another look at it? – Nat Dempkowski Jun 02 '14 at 21:49
  • 1
    That'll work. But this might not be a huge performance gain. The performance depends on how much computation each thread has to do, and how long it takes to store this data in your data structures. You might do well researching some parallel data structures and algorithms. A super simple model is the producer-consumer pattern. You can also leverage `collections.deque` which itself is threadsafe. – maxywb Jun 02 '14 at 21:57

0 Answers0