0

New to multiprocessing in python and just trying to understand why my code won't go inside the 'with Pool(processes=4) as pool: block. Here is a simple code (I removed some details because they are not necessary).

import os
import psutil
import time
import datetime
import sys
import multiprocessing
from multiprocessing.pool import Pool
import random as rnd
import math as math
from itertools import repeat

def test_func(arg1, arg2, arg3, arg4):
    do something()

if __name__ == '__main__':
    arg1 = sys.argv[1]
    arg2 = int(sys.argv[2])
    arg3 = int(sys.argv[3])
    arg4 = int(sys.argv[4])
    
    result_list = []
    print("Setting up pool")

    with Pool(processes=8) as pool:
        print("Pool is set up")
        pool.starmap_async(test_func, zip(arg1, repeat(arg2), repeat(arg3), repeat(arg4)), chunksize=3)
        pool.close()
        pool.join()
        print("The end in pool")
    print("The end in __main__")

Normally in all the circumstances I would expect it to go into the line print("pool is set up"). But occasionally it does not even go there and stops at print("Setting up pool"). Can anyone explain under what condition this might happen?

RforResearch
  • 401
  • 8
  • 16
  • `cpus_per_node` isn't defined. This code should generate a `NameError`. – Michael Ruth May 17 '21 at 18:11
  • Sorry for the confusion. I edited my answer. I had used a constant for testing. Yes it does not go inside that block – RforResearch May 17 '21 at 18:21
  • Also, `run_on_each_pdate` is not defined. I changed `test_func` to `run_on_each_pdate` with only `pass` in the body. I couldn't reproduce the behavior you describe. Please provide instructions to reproduce the behavior. – Michael Ruth May 17 '21 at 18:30
  • Question: When the code doesn't seem to execute the code in the `with` block, does it terminate on its own? – Michael Ruth May 17 '21 at 18:31
  • it does not terminate. I am running it on a cluster so it just times out – RforResearch May 17 '21 at 18:41
  • Some clusters write `stderr` and `stdout` for each job to a file. Does it throw an error? – Pranav Hosangadi May 17 '21 at 18:45
  • OK, then it could be the case that it does enter the `with` block but doesn't flush the stdout buffer before running `pool.starmap_async`. Check out [How can I flush the output of the print function?](https://stackoverflow.com/questions/230751/how-can-i-flush-the-output-of-the-print-function) for info on testing this theory. – Michael Ruth May 17 '21 at 18:47
  • On further investigation, I realized the problem was indeed with number of processes initialized. I was using processes = 1 when I needed more and it started timing out. After I fixed the input to the script, it started working fine – RforResearch May 18 '21 at 01:56

0 Answers0