Is there a better way to make multiple HTTP requests asynchronously in Ruby?

Question

I'm trying to make multiple HTTP requests in Ruby. I know it can be done in NodeJS quite easily. I'm trying to do it in Ruby using threads, but I don't know if that's the best way. I haven't had a successful run for high numbers of requests (e.g. over 50).

require 'json'
require 'net/http'

urls = [
  {"link" => "url1"},
  {"link" => "url2"},
  {"link" => "url3"}
]

urls.each_value do |thing|
    Thread.new do
        result = Net::HTTP.get(URI.parse(thing))
        json_stuff = JSON::parse(result)
        info = json["person"]["bio"]["info"]

        thing["name"] = info
    end
end

# Wait until threads are done.
while !urls.all? { |url| url.has_key? "name" }; end

puts urls

Any thoughts?

score 4 · Accepted Answer · answered Jul 13 '13 at 12:25

Instead of the while clause you used, you can call Thread#join to make the main thread wait for other threads.

threads = []
urls.each_value do |thing|
    threads << Thread.new do
        result = Net::HTTP.get(URI.parse(thing))
        json_stuff = JSON::parse(result)
        info = json["person"]["bio"]["info"]

        thing["name"] = info
    end
end

# Wait until threads are done.
threads.each { |aThread|  aThread.join }

score 2 · Answer 2 · answered Jul 13 '13 at 05:56

Your way might work, but it's going to end up in a busy loop, eating up CPU cycles when it really doesn't need to. A better way is to only check whether you're done when a request completes. One way to accomplish this would be to use a Mutex and a ConditionVariable.

Using a mutex and condition variable, we can have the main thread waiting, and when one of the worker threads receives its response, it can wake up the main thread. The main thread can then see if any URLs remain to be downloaded; if so, it'll just go to sleep again, waiting; otherwise, it's done.

To wait for a signal:

mutex.synchronize { cv.wait mutex }

To wake up the waiting thread:

mutex.synchronize { cv.signal }

You might want to check for done-ness and set thing['name'] inside the mutex.synchronize block to avoid accessing data in multiple threads simultaneously.

Is there a better way to make multiple HTTP requests asynchronously in Ruby?

2 Answers2