0

I've recently been encountering a problem with Python's multiprocessing package that has me baffled. In a program, I have the following lines of code:

import multiprocessing as mp

...

def logResult(result):
    results.append(result)

cpu = mp.cpu_count()
pool = mp.Pool(processes=cpu)

results = []
num = 400 # A tunable parameter
for x in range(0, num):
    pool.apply_async(integrationFunction, args=(<some list of arguments...>), callback=logResult)

pool.close()
pool.join()

print(len(results)) # Unpredictable

...

I cant really reveal exactly what happens in integrationFunction, but I know the following: - It uses numpy.dblquad at some point, and can be quite slow.

The problem is that in the last line of code, where I print the length of the results list, I get an erratic result every time I run the program - it usually prints some number around 100-120 when num is 400, but on odd occasions, it may print numbers outside of the range (and sometimes, even 0). But I expect it to print exactly num (which is 400 in this example) - what might be some reasons this is occurring?

Thanks!

paulinho
  • 292
  • 2
  • 13
  • Are you certain `integrationFunction` is not throwing exceptions? That's the most likely reason to me of the "erratic behaviour". – Lukasz Tracewski Oct 27 '19 at 11:19
  • I do not see any errors or warnings, and the integration function never is an exception explicitly thrown. Is it possible that some are getting suppressed in dblquad? – paulinho Oct 27 '19 at 12:21
  • 1
    `dblquad` can e.g. throw `ValueError` and number of other exceptions caused by e.g. your OS or input. How do you check for exceptions in child processes? Some guidelines you can find e.g. here: https://stackoverflow.com/questions/19924104/python-multiprocessing-handling-child-errors-in-parent – Lukasz Tracewski Oct 27 '19 at 14:16

0 Answers0