1

I've successfully applied the multiprocessing library such that it will use 12 cores on my machine to run 5,640 "jobs," each one taking about 7 minutes to complete. If I've done the math correct, this means it'll take about 78.33 hrs to complete (5,640 runs / 12 cores * ~7 min/run / 60 mins/hr).

I had a look at this post where an offered solution mutes output from a spawned python process. In my application, however, each spawned process is running a heat flow program written in Fortran. The compiled Fortran executable prints messages to the screen at the completion of each time step, of which there are 2,920 (8 years of daily time steps). When I tried implementing the mute solution offered in the post I mentioned above, it didn't stop the output from being written to the console.

I do not have the ability to recompile the Fortran code to stop it from writing msgs. Instead, is there a way to redirect the messages written by the Fortran executable to some nul space? Or to a file (if that's anticipated to be better than to the console)? Doing so should speed things up further. With 12 spawned processing all piping msgs back to the console, this is likely a bottleneck.

Here is the attempt to mute the messages coming from Fortran that didn't work:

def mute():
    sys.stdout = open(os.devnull, 'w')    


if __name__ == '__main__':
    print("Starting main...")
    with mp.Pool(12, initializer=mute) as pool:
        result = {i: v for i, v in enumerate(pool.starmap(one_model_run, product(wtT_series, infilT_series, infil_series)))}
user2256085
  • 483
  • 4
  • 15
  • Do you call subprocess Popen or run to run your Fortran code ? – Sylvaus Aug 25 '21 at 16:48
  • Does your Fortran code write to stdout or to the TTY? If you run your Fortran program in bash with `>/dev/null` or `2>/dev/null`, does that make the messages go away? (Either of those have easy Python subprocess equivalents). – Charles Duffy Aug 25 '21 at 16:51
  • Changing `sys.stdout` only effects Python code; it doesn't modify native applications; but you have plenty of choices that _do_ work with native applications. – Charles Duffy Aug 25 '21 at 16:51
  • ...and reading the proposed duplicate, it covers one of those choices (albeit a more complex one than you probably need) – Charles Duffy Aug 25 '21 at 16:52
  • ...it's likely that just setting `stdout=subprocess.DEVNULL` or `stderr=subprocess.DEVNULL` on the `subprocess.run()` or whatever your call actually is would work as well. – Charles Duffy Aug 25 '21 at 16:53

0 Answers0