How to implement pool inside a for loop

Question

I'm a college student starting to learn about multiprocessing and I was playing around with Pool. I have a version of my code that uses a map to iterate through a function called add_one from a list of numbers all under a for loop that does the steps above a certain number of times(just because).


def add_one(n):
  b = n+1
  return b

out_list = [1,2,3,4]

for x in range(3):
  oo = map(add_one, out_list)
  print(list(oo))

This version of code runs completely fine and returns 3 identical list with numbers in out_list with the addition of 1.

However, what seems to be an innocent addition of processPool to my map function gives me an wall of errors and it occurs as soon as I add process.Pool.

from multiprocessing import Pool


MAX_PROCESS = 2
processPool = Pool(processes=MAX_PROCESS)

def add_one(n):
  b = n+1
  return b

out_list = [1,2,3,4]

for x in range(3):
  oo = processPool.map(add_one, out_list)
  print(list(oo))

The error message:

Process ForkPoolWorker-2:
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'add_one' on <module '__main__' from 'main.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'add_one' on <module '__main__' from 'main.py'>

I've tried reading the error message, but can't seem to grasp why it's happening. Am I just not allowed to use the pool inside a for loop?

I think all you need to do is add the `__name__ == '__main__':` bit. That worked for me [here](https://ideone.com/BrLT3F) — Paul Rooney, Nov 27 '20 at 04:46

How to implement pool inside a for loop

0 Answers0