Limiting concurrent threads

Question

I'm using threads in a program that uploads files over sftp. The number of files that could be upload can potentially be very large or very small. I'd like to be able to have 5 or less simultaneous uploads, and if there's more have them wait. My understanding is usually a conditional variable would be used for this, but it looks to me like that would only allow for 1 thread at a time.

cv = ConditionVariable.new

t2 = Thread.new {
  mutex.synchronize {
    cv.wait(mutex)
    upload(file)
    cv.signal
  }
}

I think that should tell it to wait for the cv to be available the release it when done. My question is how can I do this allowing more than 1 at a time while still limiting the number?

edit: I'm using Ruby 1.8.7 on Windows from the 1 click installer

Ruby version you are using is important for this question. – Michael Papile Jun 14 '11 at 15:17 — Michael Papile, Jun 14 '11 at 15:17

score 2 · Accepted Answer · edited May 23 '17 at 11:55

2

Use a ThreadPool instead. See Deadlock in ThreadPool (the accepted answer, specifically).

edited May 23 '17 at 11:55

Community

1
1

answered Jun 14 '11 at 15:19

coreyward

77,547
20
137
166

score 1 · Answer 2 · answered Jun 14 '11 at 16:28

1

A word of caution -- there is no real concurrency in Ruby unless you are using JRuby. Also, exception in thread will freeze main loop unless you are in debug mode.

require "thread"

POOL_SIZE = 5

items_to_process = (0..100).to_a

message_queue = Queue.new

start_thread = 
  lambda do
    Thread.new(items_to_process.shift) do |i|
      puts "Processing #{i}"
      message_queue.push(:done)
    end
  end

items_left = items_to_process.length

[items_left, POOL_SIZE].min.times do
  start_thread[]
end

while items_left > 0 
  message_queue.pop
  items_left -= 1
  start_thread[] unless items_left < POOL_SIZE
end

answered Jun 14 '11 at 16:28

Victor Moroz

9,167
1
19
23

1

This is not true. Version 1.9.2 of MRI uses a native/kernel threads with a GIL. It's not as effective as the JRuby or MacRuby threads, but they aren't the green threads that 1.8.7 has. In the case of doing IO work (like the question implies) 1.9.2 threads are quite useful (just look at Typhoeus if you don't believe me). – coreyward Jun 14 '11 at 18:11
@coreyward Agree. I have always thought that even in 1.8.7 IO is in fact done in parallel. Hard to believe that Ruby uses blocking calls to IO system that freeze scheduler. Besides, thread pools can be used to limit socket usage, file handles etc. – Victor Moroz Jun 14 '11 at 18:50

Limiting concurrent threads

2 Answers2