1

I need to run 20 experiments in parallel. For each, I am loading a new unix screen, then within that loading an anaconda environment, and then running a python script with an argument for that experiment.

The workflow looks like:

> screen -S exp01
> source activate myenv
(myenv) > python process_experiment 01
> screen -S exp02
> source activate myenv
(myenv) > python process_experiment 02

Is there a way to write a script to automate it?

Adam_G
  • 7,337
  • 20
  • 86
  • 148

1 Answers1

1

According to this question, you can send keystrokes to a screen session like so:

screen -dmS new_screen sh
screen -S new_screen -X stuff "cd /dir
"
screen -S new_screen -X stuff "java -version
"

You could write a small shell script (let's call it run-exp.sh) to set your experiment running:

#!/bin/sh
source activate myenv
python process_experiment $1

And then use a loop to start screen sessions running that script:

#!/bin/sh
for i in $(seq -w 1 20); do
    screen -dmS exp$i
    screen -S exp$i -X stuff "./run-exp.sh $i
"
    # (The new line is necessary, not a mistake.)
done
Harry Cutts
  • 1,352
  • 11
  • 25
  • I just realized an error in my description: the experiment number needs to be 2 digits, so 1 needs to be 01 – Adam_G Aug 31 '18 at 15:03
  • Oops, this does '01' instead of 1, right? In any case, I'm not seeing anything in any screen – Adam_G Aug 31 '18 at 15:09
  • Yes, the `-w` flag makes `seq` pad the lower numbers with zeros. Hmm. Not totally sure how to debug; are the screen sessions being created correctly? – Harry Cutts Aug 31 '18 at 18:23
  • Yes, there are 20 screen sessions that I see listed. But nothing is happening in them. – Adam_G Aug 31 '18 at 18:25
  • So another way of executing a command within the session is to pass it as an argument to the shell when you create the shell. So, in this case, inside the `for` loop try just `screen -dmS exp$i bash -c "./run-exp.sh $i"`. – Harry Cutts Aug 31 '18 at 19:47
  • Correction: "…when you create the _session_." – Harry Cutts Aug 31 '18 at 20:02
  • hmmm, still not doing anything – Adam_G Aug 31 '18 at 22:02