Python Passing Function Handles To Threads

Question

I'm trying to call 2 functions in parallel using the python 2.7 threading module, and, to make things easier for myself, I've written my own AsycTask class.

class AsyncTask:
    def __init__(self, task, name="async-task", callback=None):
        self.task = task
        self.t = None
        if self.task is not None:
            if callback is not None:
                self.t = threading.Thread(target=lambda: [task(), callback()], name=name)
            else:
                self.t = threading.Thread(target=self.task, name=name)

    def start(self):
        if self.t is not None:
            self.t.start()
        else:
            Log.warn("Can't start async task: thread is None")

    def join(self):
        if self.t is not None:
            self.t.join()
        else:
            Log.warn("Can't join async task: thread is None")

But I get some strange results when I pass it function handles.

Elsewhere I have this class:

class Foo:
    def __init__(self, id):
        self.id = id

    def bar(self, args):
        result = None
        # do some stuff that takes a while
        time.sleep(10)

        Log.debug("bar() called in object %s" % self.id)
        return result

Then I create a list of foo's

foos = []
foos.append(Foo("1"))
foos.append(Foo("2"))

And call bar asynchronously

results = []
tasks = []
for foo in foos:
    args = "some stuff"
    fn = foo.bar
    Log.debug("before async: " + str(foo))

    task = AsyncTask(lambda: [Log.debug("in async: " + str(fn)), results.append(fn(args))])
    tasks.append(task)
    task.start()

for task in tasks:
    task.join()

# process results

When I run this I get:

before async: <__main__.Foo instance at 0x7f9caef7e200>
before async: <__main__.Foo instance at 0x7f9caef7e248>
in async: <bound method Foo.bar of <__main__.Foo instance at 0x7f9caef7e248>>
in async: <bound method Foo.bar of <__main__.Foo instance at 0x7f9caef7e248>>
bar() called in object 2
bar() called in object 2

Note that bar() on the first Foo instance is never called, while it is called twice on the second instance.

I don't have a ton of python experience, so clearly, I am doing something wrong and must not properly understand how threading and function handles work in python.

What would a more pythonic way be to accomplish this?

This has nothing to do with threading; this is an issue with you passing in a `lambda` using `fn` as a closure. — Martijn Pieters, May 13 '18 at 10:36
@MartijnPieters can you elaborate or link to some info about what you mean? What is a better solution? — JohannB, May 13 '18 at 14:14
I closed this post as a duplicate, see the other post, which outlines several options. — Martijn Pieters, May 13 '18 at 14:16

score 0 · Answer 1 · answered May 10 '18 at 15:52

0

Your setup classes are fine, the reason you are getting the odd behavior which we can't see is most likely how you set up your logger. I got the same behavior you did until I setup my logger as follows:

import logging

logging.basicConfig()
log = logging.getLogger('log')

Afterwards I get the following:

before async: <__main__.Foo instance at 0x00000000028C9FC8>
before async: <__main__.Foo instance at 0x00000000028D5048>
<__main__.Foo instance at 0x00000000028C9FC8>
<__main__.Foo instance at 0x00000000028D5048>

answered May 10 '18 at 15:52

user1321988

513
2
6

Thanks for the quick reply, but I'm not convinced. In my actual setup, `Foo` is an interface to some hardware, and `bar()` is definitely only getting called on one instance. – JohannB May 10 '18 at 16:08
I've edited my question by adding an `id` class-member to 'Foo` to illustrate the problem better – JohannB May 10 '18 at 16:17

Python Passing Function Handles To Threads

1 Answers1