How to launch in Ruby commands in parallel

Question

I am writing a terminal program using ruby as a launcher for a set of program written in C++ and Java which should be executed in a distributed system.

I would like to translate this instruction in ruby:

for i in {1..40}; do
  ssh node$i program & #note & so that that process is detached
done

This is my ruby code:

class Launcher
   # Other method that we can ignore
   def order(command)
     @nodelist.each {#Do something here}
   end 
end

I though about creating a pool of thread and each thread execute that command. Is it the appropriate way? As I studied threads can not execute "exec" since threads share the same memory address.

Do you want your Ruby process to still be alive after it launches everything? Does the Ruby process have to do anything with the output of these commands or wait for them to finish? — Andrew Marshall, Feb 22 '15 at 15:58
Each process has at least one thread. As far as I know, all threads of a process are able to do `exec`. Though `exec()` replaces one process by another. That means that all the threads that belong to the old process are gone. — hagello, Mar 01 '15 at 21:15

score 6 · Accepted Answer · answered Feb 23 '15 at 11:36

Here is the solution:

class Launcher

  def initialize(commands_list)
    #Expected to be an array of command string
    @commands_list = commands_list
  end

  def execute
    p_threads = @commands_list.map do |command|
      Process.detach(Process.spawn(command))
    end
    p_threads.each(&:join)
   end
end

score 1 · Answer 2 · answered Mar 01 '15 at 21:28

1

Do you know GNU parallel? It's made for running jobs in parallel (surprise!). It is developped by experienced people, tested and tried. You might just use it instead of reimplementing most of it. Or you might have a look in its manual and/or source code, and maybe learn from it.

answered Mar 01 '15 at 21:28

hagello

2,843
2
27
37

Does it have a ruby library? Or are you refering about launching gnu parallel using the backquotes? – Vicente Bolea Mar 02 '15 at 00:25
Can you explain how to extend it to a cluster? – Vicente Bolea Mar 02 '15 at 11:43
@Vicente After looking around a bit, I do not think that you can use GNU parallel as a library. You can call it using backquotes, `system` or `Popen3`. I do not have any experience in using it on clusters, but you will want to have a look at http://stackoverflow.com/questions/22236337/gnu-parallel-jobs-option-using-multiple-nodes-on-cluster-with-multiple-cpus-pe – hagello May 05 '15 at 18:14

How to launch in Ruby commands in parallel

2 Answers2