I have a function that sequentially searches for a particular feature in a list of image files, and terminates as soon as it finds a match. The function takes almost a second to run so I was hoping to employ multiprocessing to see if I can speed it up.
def match(a, b, feature) -> boolean:
# time.sleep([0 - 10]) # simulates a long time-consuming function
# return isMatch # could be True or False depending on whether feature was found
p = Pool(processes=4)
list = p.map(match, [
['imgA', 'img1', feature],
['imgA', 'img2', feature],
['imgA', 'img3', feature],
['imgA', 'img4', feature],
['imgA', 'img5', feature],
...
...
['imgA', 'img100', feature],
])
found = next(x for x in list if x == True, False)
This works fine. BTW I am not actually hard-coding the parameters array. This is just for illustration purposes. However, I have a couple of questions regarding this approach:
Right now, control doesn't return back to me until all 100 processes have completed execution, even if a match was found very early on. Is there a way I could signal from one of the child processes that it has succeeded so that the parent process can terminate all spawned processes and remove any pending processes in the queue?
If that is possible, I would like to ensure that if that parent process was itself spawned, a child process experiencing success would not kill the parent's sibling process's child processes as well. In this example, I have kept
a
andfeature
constant. I would like to expand this solution so I can have botha
andb
as lists. In that case, for eacha
, I would spawn a child process to process a singlea
against all possibleb
s, which in turn should spawn grand-children that would process a singlea
against a singleb
. Every time I find a match between ana
and ab
, I would like to terminate all grand-children for the child process that was processing a singlea
against all possibleb
s.
If someone can guide me in the right direction, or even a library that gets this done for me, I would be really grateful. =)
Thanks a lot, Asim