Python Multiprocessing to invoke a class function

Question

I am trying to multiprocess a large number of objects that I need to invoke a class function off. I cant pull the function outside of the class and so just need to figure out a way to speed up the calling of this function.

Many thanks in advance

import time

# - Bespoke Class that is standalone - #
class sub_class:
    def __init__(self, item_id, assumptions = "BlahBlahBlah"):
        self.item_id = item_id
        self.assumptions = assumptions
        self.test_count = 0

    def some_func(self, adder=1):
        self.test_count += adder
        time.sleep(0.1)


# - initialise a list of classes - #
item_ids = ['A', 'B', 'C', 'D']
items = [sub_class(_id) for _id in item_ids]

print( sum(_item.test_count for _item in items) )
# Prints out 0

# - invoke a class level function - #
[_item.some_func(adder=2) for _item in items]

print( sum(_item.test_count for _item in items) )
# Prints out 8

Does this answer your question? [python Pool with worker Processes](https://stackoverflow.com/questions/9038711/python-pool-with-worker-processes) — stovfl, Feb 27 '20 at 12:12

FiddleStix · Accepted Answer · 2020-02-27T17:14:09.627

This works with only a minor change to some_func().

import time
from concurrent.futures import ProcessPoolExecutor, wait, ALL_COMPLETED


# - Bespoke Class that is standalone - #
class MyClass:
    def __init__(self, item_id, assumptions="BlahBlahBlah"):
        self.item_id = item_id
        self.assumptions = assumptions
        self.test_count = 0

    def some_func(self, adder=1):
        self.test_count += adder
        print(f"{self.item_id} executing some_func()")
        time.sleep(0.1)
        # note: we must return ourselves
        return self


# Don't omit this on Windows
if __name__ == '__main__':

    # - initialise a list of classes - #
    item_ids = ['A', 'B', 'C', 'D']
    items = [MyClass(item_id) for item_id in item_ids]

    print(sum(item.test_count for item in items))
    # Prints out 0

    # - invoke a class level function - #
    with ProcessPoolExecutor(max_workers=4) as executor:
        future_list = []
        for item in items:
            future_list.append(executor.submit(item.some_func, adder=2))

        wait(future_list, return_when=ALL_COMPLETED)

        # note: we must use the returned self to get the test count
        print(sum(future.result().test_count for future in future_list))
        # Prints out 8

        # note: the original items haven't changed
        print(sum(item.test_count for item in items))
        # Prints out 0

The "items" get copied to the new Process so you have to use the future return() values instead. Therefore, you can replace the original items using items = [future.result() for future in future_list] at some point after the wait().

This works perfectly, exactly what I was looking for. ( Ive add a little progress bar and it work like a charm) — user3112131, Feb 27 '20 at 15:49
Perhaps, I shouldn't have been as hasty: how would you store the new outputs? e.g. Doing the below prints 0 tmp_list = [future.result().test_count for future in future_list] print(sum(tmp_list)) — user3112131, Feb 27 '20 at 17:10
Apologies, this works; I had adapted a few things and left out the "return self" bit in MyClass.some_func (despite your comment not to forget it). thanks a lot — user3112131, Feb 27 '20 at 23:22

Python Multiprocessing to invoke a class function

1 Answers1