ValueError: 'Pool not running' when trying to loop the multiprocessing pool

Question

I'm trying to run the same pool three times so that it adds the values in the same list:

import multiprocessing

def foo(name,archive):
    print('hello ', name)
    archive.append(f"hello {name}")

def main():
    max_process = multiprocessing.cpu_count()-1 or 1
    pool = multiprocessing.Pool(max_process)
    manager = multiprocessing.Manager()

    archive = manager.list()

    arguments = ['home','away','draw']
    for _ in range(3):
        [pool.apply_async(foo, args=[name,archive]) for name in arguments]
        pool.close()
        pool.join()
    print(archive)

if __name__ == '__main__':
    main()

The first batch runs perfectly, but when the second batch goes, this error appears:

ValueError: Pool not running

How should I proceed to generate this looping?

you close the pool after the first iteration and then you try to join a closed pool. move the join to be above the close and move both of them out of the loop — Nullman, May 03 '22 at 11:49
Hi @Nullman if I put ```join``` above ```close```, this error appears → ```ValueError: Pool is still running``` but keeping ```close``` above ```join``` worked perfectly outside the loop. But is it wrong to use it this way? — Digital Farmer, May 03 '22 at 12:01
read [this](https://stackoverflow.com/q/38271547/7540911) question and answers for more details — Nullman, May 03 '22 at 12:04
Thanks for the link, but I still have doubts, I saw the usefulness of ```join``` being called above ```close```, but for my case the error ```ValueError: Pool is still running``` keeps appearing whenever I try to use this way. I don't understand how I can solve this detail. — Digital Farmer, May 03 '22 at 12:26
it seems like you need to do it the other way around with multiprocessing, call close first and then join — Nullman, May 03 '22 at 12:30

score 1 · Accepted Answer · answered May 03 '22 at 12:41

As indicated by Nullman in comments, the error was in keeping pool.close() and pool.join() inside the loop, bringing them out, worked perfectly:

import multiprocessing

def foo(name,archive):
    print('hello ', name)
    archive.append(f"hello {name}")

def main():
    max_process = multiprocessing.cpu_count()-1 or 1
    pool = multiprocessing.Pool(max_process)
    manager = multiprocessing.Manager()

    archive = manager.list()

    arguments = ['home','away','draw']
    for _ in range(3):
        [pool.apply_async(foo, args=[name,archive]) for name in arguments]
    pool.close()
    pool.join()
    
    print(archive)

if __name__ == '__main__':
    main()

Well, thats the sollution, but why even use join then if it can't be used inside loop for waiting for all processes — Grzegorz Krug, May 04 '23 at 20:27

ValueError: 'Pool not running' when trying to loop the multiprocessing pool

1 Answers1