Run scripts in parallel in ruby

Question

I need to convert videos in 4 threads

For example I have Active Record models Video with titles: Video1, Video2, Video3, Video4, Video5

So, I need to execute something like this

bundle exec script/video_converter start

Where script will process unconverted videos for 4 threads, for example

Video.where(state: 'unconverted').first.process

But if one of 4 videos are converted, next video must be automatically added to thread

What is the best solution for this ? Sidekiq gem? Daemons gem + Ruby Threads manually?

For now I am using this script:

THREAD_COUNT = 4
SLEEP_TIME = 5
logger = CONVERTATION_LOG
spawns = []
loop do
  videos = Video.where(state:'unconverted').limit(THREAD_COUNT).reorder("ID DESC")
  videos.each do |video|
    spawns << Spawnling.new do
      result = video.process
      if result.nil?
        video.create_thumbnail!
      else
        video.failured!
      end
    end
  end
  Spawnling.wait(spawns)
  sleep(SLEEP_TIME)
end

But this script waits 4 videos, and after it takes another 4 videos. I want, that after one of 4-th video converted, it will be automatically added to new thread, which is empty.

Threads aren't parallel on some runtimes, e.g. MRI. What do you use? — sschmeck, Dec 16 '15 at 13:28
Even with MRI it's not quite as simple, e.g. if calling through to native code that releases the GVL or spawning an ffmpeg process — Frederick Cheung, Dec 16 '15 at 13:32
I am using [spawnling gem](https://github.com/tra/spawnling), I am added additional example to first message — rs41, Dec 16 '15 at 17:11

score 1 · Answer 1 · edited May 23 '17 at 10:27

1

If your goal is to keep processing videos by using just 4 threads (or whatever Spawnling is configured to use - as it supports fork and thread), then, you could use a Queue to queue all your video records to be processed, spawn 4 threads and let them keep processing records one by one until queue is empty.

require "rails"
require "spawnling"

# In your case, videos are read from DB, below array is for illustration
videos = ["v1", "v2", "v3", "v4", "v5", "v6", "..."]


THREAD_COUNT = 4

spawns = []

q = Queue.new 

videos.each {|i| q.push(i) }

THREAD_COUNT.times do
  spawns << Spawnling.new do
    until q.empty? do
      v = q.pop

      # simulate processing
      puts "Processing video #{v}"

      # simulate processing time
      sleep(rand(10))
    end
  end
end

Spawnling.wait(spawns)

This answer is inspired from this answer

_{PS: I have added few requires and defined videos array to make above code self-contained running example.}

edited May 23 '17 at 10:27

Community

1
1

answered Dec 16 '15 at 18:05

Wand Maker

18,476
8
53
87

1

And if you don't want to load all your unprocessed videos at once, set up a 5th thread that sleeps until the queue is below some threshold, then refills it. – Kristján Dec 16 '15 at 18:20
@Wand Maker, thanks's a lot for suggestion ! 2 notices: 1) For your logic , if we have this array `["v1", "v2", "v3", "v4", "v5", "v6"]` The script add `["v1", "v2", "v3", "v4", "v5", "v6"]` to first thread, and none to other second, third and fourth thread, because untill is a loop with condition But I need that to first thread added "v1", to second "v2", to 3-th "v3", to 4-th "v4", to first "v5", to second "v6" and etc – rs41 Dec 17 '15 at 07:29
2) It looks, that your example added videos at once, and after that process whole array But sometimes, manager of project periodically add new videos, and I want to add this videos to Existing queue, not to new queue( So videos must async added to queue and script must be add this videos to current 4 threads). What you think about it? – rs41 Dec 17 '15 at 07:29
@rs41 I think each thread pops only one video at a time - so all 4 threads (once they are running) should be pulling videos one by one. The `Queue` class provides a way to synchronize communication between threads. – Wand Maker Dec 17 '15 at 07:34

Run scripts in parallel in ruby

1 Answers1