1

I used python(2.7) module multiprocessing, to implement parallel processing in one of my scripts.

import sys
import multiprocessing

def myfun(file):
  try:
      <my logic>
  except Exception:
    sys.exit(-1)

my_file_list ['file1', 'file2', 'file3', 'file4', 'file5', 'file6', 'file7', 'file8', 'file9', 'file10', 'file11', ]
pool = multiprocessing.Pool(processes=3) 
pool.map(myfun, my_file_list )

Whenever my function myfun error out because of some file anomalies, my main function will go into a infinite wait! and I wi;; have to kill it. Can someone help me to fix my code/logic?

L3viathan
  • 26,748
  • 2
  • 58
  • 81
Turbo Sullivan
  • 827
  • 1
  • 7
  • 9
  • What kind of exception is raised in ? This might have to do with https://bugs.python.org/issue9400 – L3viathan Jul 31 '18 at 07:21
  • Possible duplicate of [cancel join after sys.exit in multiprocessing](https://stackoverflow.com/questions/23934399/cancel-join-after-sys-exit-in-multiprocessing) – zwer Jul 31 '18 at 07:27
  • Getting error like this "(InvalidRange) when calling the GetObject operation: The requested range is not satisfiable" – Turbo Sullivan Jul 31 '18 at 07:27
  • You can also check this question. [Python Multiprocessing: Handling Child Errors in Parent](https://stackoverflow.com/questions/19924104/python-multiprocessing-handling-child-errors-in-parent) – ganit44 Jul 31 '18 at 07:28

1 Answers1

0

Don't call sys.exit unless you have a very good reason and know what you are doing: it's the pool's responsibility to manage the process. It looks like you don't need that try-except logic at all.

Manu Valdés
  • 2,343
  • 1
  • 19
  • 28