8

I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread

snippet of the code:

cp = ConfigParser.ConfigParser()
cp.read(gdbini)
for table in cp.sections():
    jobs.append(table)
#print jobs
poolreturn = pool.map(worker, jobs)
pool.close()
pool.join()

Failure Message:

Traceback (most recent call last):
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/cnet-python/default-2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'ConfigParser.NoOptionError'>, ("No option 'inputfilename' in section: 'section-1'",))

I went ahead added a exception handler to terminate the process 

try:
    ifile=cp.get(table,'inputfilename')
except ConfigParser.NoSectionError,ConfigParser.NoOptionError:
    usage("One of Parameter not found for"+ table)
    terminate()

but still it waits, not sure whats missing.

Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
mk.
  • 81
  • 1
  • 2
  • Looks like ConfigParser has the same problem as SQLAlchemy (exceptions not pickleable), see [Hang in Python script using SQLAlchemy and multiprocessing](http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing). I've reported this issue as [ConfigParser exceptions are not pickleable](http://bugs.python.org/issue13760). – Faheem Mitha Jan 11 '12 at 07:56

2 Answers2

2

In Python 3.2+ this works as expected. For Python 2, this bug was fixed in r74545 and will be available in Python 2.7.3. In the mean time, you can use the configparser library which is a backport of the configparser from 3.2+. Check it out.

Łukasz Langa
  • 31
  • 3
  • Was this a big in configparser or multiprocess? (I'm getting the same error with multiprocess, unrelated to configparser) – user48956 Oct 11 '17 at 22:00
0

I had the same issue. It happens when a worker process raises a user exception which has a custom constructor. Make sure your exception (ConfigParser.NoOptionError in that case) initializes the base exception with exactly two arguments:

class NoOptionError(ValueError):

    def __init__(self, message, *args):
        super(NoOptionError, self).__init__(message, args)