2

This is a continuation of this question and this question on nested GNU Parallel. Utlimately what I want to achieve is to leave my Makefile untouched except by changing the SHELL= variable and distribute jobs using parallel across all my machines.

Is there a way to ensure that concurrent executions of GNU Parallel respect the --jobs clause specified in the outer invocation? Or some other way to get a limit on the total number of jobs across parallel invocations? For example: I'd like the inner slot in the output below to always be 1, i.e., the slot 1-2 on line three of the output violates the condition.

~• inner_par="parallel -I // --slotreplace '/%/' --seqreplace '/#/'"
~• cmd='echo id {#}-/#/, slot {%}-/%/, arg {}-//'
~• seq 2 | parallel -j 1 "seq {} | $inner_par $cmd"
id 1-1, slot 1-1, arg 1-1
id 2-1, slot 1-1, arg 2-1
id 2-2, slot 1-2, arg 2-2
~•
Leo
  • 2,775
  • 27
  • 29

1 Answers1

2

Are you looking for sem?

parallel -j 10 parallel -j 20 sem -j 30 --id myid mycmd

This will start 200 sems, but only run 30 mycmds in parallel.

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • Yes, Ole! Sem was helpful. But unfortunately it deletes the semaphore after it is done. I wanted an option to point to the semaphore file so it could be shared between different users. Is that possible/practical? – Leo Jul 03 '20 at 01:16
  • Hmm... No. Currently that is not possible. A semaphore is limited to a single user. – Ole Tange Jul 03 '20 at 08:06
  • Another way to achieve what I want is to start Parallel following the "simple queue" example from the doc. The trouble there is capturing the exit code of the right job and making the `SHELL=` wrapper wait for that specific job to complete. I'm looking into doing a `wc` to get the job id, but that again requires a shared semaphore. Furthermore, when a job fails and is restarted does it keep the same JobId? – Leo Jul 05 '20 at 12:39
  • 1
    That might work. But use `PARALLEL_SHELL=` instead of `SHELL=`. JobIDs remain the same. – Ole Tange Jul 05 '20 at 12:47