Multiprocessing pool.map does not return from function call

Question

I have a short snippet making the pool.map call to hang up. It does not return in any reasonable amount of time. The idea is to have a couple of point lists and find a measure for the correspondence to another point list.

    import numpy as np
    import multiprocessing
    import scipy.spatial as ss

    def dist(args): # returns the sum of distances of the nearest neighbors.
        kdSet, points = args
        dist, _ = kdSet.query(points)
        return np.sum(dist)

    #just define some points to play with
    points = np.array([[13.27, 25.49], [13.18, 25.39], [13.08, 25.39], 
                       [12.99, 25.39], [12.89, 25.39], [12.80, 25.39],
                       [12.71, 25.39], [12.61, 25.30], [12.52, 25.30]]) 
    pointList = [points + idx for idx in range(3)] #creates a list of points

    #create a list of KDTrees for fast lookup. Add some number to make them a little different from pointList
    kdTreeList = [ss.KDTree(pointsLocal + idx/10) for idx, pointsLocal in enumerate(pointList)] 

    #this part works fine: serial execution
    distances = list(map(dist, zip(kdTreeList, pointList)))
    print(distances)

    #this part hangs, if called as multiprocess, expected the same result as above
    with multiprocessing.Pool() as pool:
        print('Calling pool.map')
        distancesMultiprocess = pool.map(dist, zip(kdTreeList, pointList)) #<-This call does not return, but uses the CPU 100%
    print(distancesMultiprocess)

Calling the function sequentially it works fine. But if called in the pool.map function it neither gives an error nor it is returning anything. It just hangs there. The CPU load goes to 100%, so something is happening.

Removing the KDTree class for testing purposes did not show a different result.

Since there is no error thrown, I have no idea whats going wrong here. Any hint?

Your code works fine for me, using python-3.5. I remember that older python (<=3.3 ?) had issues with the `with ... as pool` syntax. Could it be a clue? — Demi-Lune, Feb 14 '20 at 12:30
Are you on windows? You need to use an `if __name__ == "__main__"` guard on windows, or else you will create a multiprocessing bomb — juanpa.arrivillaga, Feb 14 '20 at 12:34
You may also try `pool.join() ; pool.close()`. Read for instance https://stackoverflow.com/questions/20387510/proper-way-to-use-multiprocessor-pool-in-a-nested-loop or https://stackoverflow.com/questions/38271547/when-should-we-call-multiprocessing-pool-join — Demi-Lune, Feb 14 '20 at 12:34

score 0 · Answer 1 · answered Feb 14 '20 at 12:39

0

Thanks a lot to juanpa.arrivillaga adding

    if __name__ == "__main__":

did the trick.

answered Feb 14 '20 at 12:39

B. Biehler

101
8

Multiprocessing pool.map does not return from function call

1 Answers1