3

I have a program which I want to create N instances of, where the only thing that varies is some hyper parameter $\beta$.

In my mind I know I could do this with a bash script, where I call the program N times, each with a different value for $\beta$, and send each one to the background so that the next one can run:

#!/bin/bash

nohup python3 test.py 1 >> res.txt &
nohup python3 test.py 2 >> res.txt &
nohup python3 test.py 3 >> res.txt &
nohup python3 test.py 4 >> res.txt &

Maybe I can also do this directly in python, in a cleaner manner. My question is, from your experience, what is the cleanest way of achieving this? Feel free to ask any detail I might have missed.

Miguel
  • 1,295
  • 12
  • 25

2 Answers2

2

For running multiple things in parallel, the thing that comes to my mind is GNU Parallel.

So for your example, a dry-run gives this:

parallel --dry-run 'nohup python prog.py {} &' ::: {1..4}

Sample Output

nohup python prog.py 3 &
nohup python prog.py 2 &
nohup python prog.py 1 &
nohup python prog.py 4 &

In general, you don't want multiple, parallel processes writing to the same file - it makes a mess, so I would name the output file after the parameter:

parallel --dry-run 'nohup python prog.py {}  > res{}.txt &' ::: {1..4}
Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • What does the flag --dry-run do exactly? It says "Print the job to run on stdout (standard output), but do not run the job". If you use it, it doesn't run, just for show? – Miguel May 19 '19 at 13:38
  • Yes, that’s exactly it. It is for testing and debugging and it means *”show what the command would do without actually doing anything”*. – Mark Setchell May 19 '19 at 14:05
  • Unless you know they are required, then `nohup` and `&` are not needed. So also try without those and see if the command does what you intended. – Ole Tange May 20 '19 at 06:14
  • @OleTange Yes, I agree. I assumed OP wants jobs to stay running after he logs out, but if that is not the case `nohup` and `&` can be dropped. – Mark Setchell May 20 '19 at 06:46
1

You are looking for the subprocess module.

subprocess.run([process_name, arg1, arg2, argn])

An Example.

import subprocess

subprocess.run(["ls", "-l"])

Also check how to call a subprocess and get the output

babaliaris
  • 663
  • 8
  • 16
  • But I would have to do that in a different ".py" file right? – Miguel May 19 '19 at 09:31
  • No. If you want let's say run 4 different processes called program1 - program4 just call subprocess.run() 4 times with the different names. – babaliaris May 19 '19 at 13:15
  • Yes, I meant in a different file from the original code, instead of doing it from inside the file. Like, "run the code below in 4 processes", probably you can also do that but needs more text – Miguel May 19 '19 at 13:35
  • @Miguel I can't quite understand your question but I will give an answer based on what I think you're asking. **The subprocess module allows you to spawn new processes** . I believe you think that because you are calling .run() in the same file all that programs will run on the same process? NO. New processes will get spawned and run each program independent from the other processes. – babaliaris May 19 '19 at 13:48