10

I'm pulling multiple requests (its pulling one at a time), I was wondering if there's a way pull requests all at the same time if I have something like this:

client = Instagram.client(:access_token => session[:access_token])
@user = client.user
@recent_media_items = client.user_recent_media

@lv = client.tag_recent_media('lv', options = {:count => 60})
@lv1 = client.tag_recent_media('lv1', options = {:count => 60})
@lv2 = client.tag_recent_media('lv2', options = {:count => 60})
@lv3 = client.tag_recent_media('lv3', options = {:count => 60})

Each lv makes a request to client. I was wondering if there's a way to do such so it can do the request all at once (together), instead of one finishes the request, then goes on to the next request, and so on...

Thanks!

hellomello
  • 8,219
  • 39
  • 151
  • 297

2 Answers2

14

Yes! For a proof-of-concept, try

require 'thread'

client = Instagram.client(:access_token => session[:access_token])
@user = client.user
@recent_media_items = client.user_recent_media

threads = []
threads << Thread.new { @lv = client.tag_recent_media('lv', options = {:count => 60}) }
threads << Thread.new { @lv1 = client.tag_recent_media('lv1', options = {:count => 60}) }
threads << Thread.new { @lv2 = client.tag_recent_media('lv2', options = {:count => 60}) }
threads << Thread.new { @lv3 = client.tag_recent_media('lv3', options = {:count => 60}) }
threads.each(&:join) # this waits for all the threads to finish before proceeding
puts [@lv, @lv1, @lv2, @lv3]

In practice, you will want to set up some error handling and retry settings within the threads. Also, you may run into issues with thread-safety in the Instagram gem. If you are doing this on a large scale with hundreds or thousands of requests, you may want to try a concurrent HTTP client like Typhoeus or an evented HTTP client like EM-HTTP-Request. For these, you would have to manually implement the tag_recent_media method that is in the Instagram gem.

Benjamin Manns
  • 9,028
  • 4
  • 37
  • 48
  • holy moley, i think this one worked! hmm I guess it should be fine if I just did this in production right? – hellomello Apr 18 '13 at 06:49
  • 3
    Yes, but you probably would want some error handling. If an exception happens during one of the threads, `threads.each(&:join)` will raise an exception. From there, you can use [Thread#status](http://ruby-doc.org/core-2.0/Thread.html#method-i-status) to check which threads had exceptions. You can also use something like [retryable](https://github.com/nfedyashev/retryable) within the thread blocks to handle networking errors. – Benjamin Manns Apr 18 '13 at 12:57
  • Isn't your example lacking a `threads <<` in front of each `Thread.new {...}`? – Patrick Oscity May 31 '13 at 10:54
5

There are gems like spawn that do this in a forked process or as a new thread in your process.. Here is a link to another SO post that discusses it

What is the difference between forking and threading in a background process?

Here is the gem in question: https://github.com/tra/spawnling

Community
  • 1
  • 1
John Armstrong
  • 158
  • 7