1

In a very crowded hpc cluster where the job submission queues are very long, I want to evaluate the performance of an algorithm (written in R) which depends on a parameter list. There are 5 possible settings for this parameter list. When I submit a new job it will queue for some unknown time (depending on the jobs submitted before mine) before running rather than start running immediately.

Is there a way for me to submit the five jobs automatically and under version control, in the sense that I do not need to wait and constantly check whether the queuing job starts running so that I can modify the values, git push, and submit a another job with new parameter list? Also I want 5 jobs be in the queue as fast as possible.

My idea is to submit 5 jobs using 5 scripts for the 5 parameter list. But this will involve lots of copy paste, does not scale well when there are say 20 possible values for the parameter list. And it does not work well with git.

Update This post is a duplicate. Thanks @High Performance Mark and @atalbot for pointing out. I will learn more on design patterns.

Thanks!

kevin
  • 174
  • 6
  • 1
    Your mention of version control, and of copy and paste, makes me worry that you change the parameterisation of your code by modifying its source. Rather than by reading the parameters from the command line or from an input file. Am I misreading the question ? – High Performance Mark Mar 15 '19 at 11:36
  • Thanks @High Performance Mark ! I have functions and main file. The parameters are defined in main file because there are more than 10 parameters that specify which algorithm to use, and if a particular algorithm is chosen, which parameters to use for that algorithm. How could I have designed it better? – kevin Mar 16 '19 at 06:40
  • I'm still not entirely sure I understand your code. But if the way you change the 10 parameters in the main file is that you edit the text of that file, then a far better design would be a code which reads the values of the parameters when it starts, either from a command line or from a parameter file. You think that your question is not a duplicate of the one @atalbot refers you to, but it should be. – High Performance Mark Mar 16 '19 at 13:17
  • Thanks @High Performance Mark, is there any introductory design books that you can recommend? – kevin Mar 16 '19 at 14:11

0 Answers0