0

First time learning about concurrency and threading within Rails, so any advice is very appreciated.

I currently have an array of 50 strings. I have an 3rd party API call that takes in the string and returns a numeric value. Right now I am simply calling the API on each string one at a time, which takes a really long time.

After looking at a few SO like this one, this other one and finally this one, it seems like I have to use some sort of threading to achieve what I want to do. My plan is to break down the array into batches of ten strings, and then run 5 API calls on each array of ten strings concurrently in hopes that it will drastically reduce the time.

I've never done threading of any kind with rails before, so I just wondering if I am on the right track following the third SO post above, or if I should use other techniques that may be better for my need.

Community
  • 1
  • 1
user3277633
  • 1,891
  • 6
  • 28
  • 48

1 Answers1

0

The approach you take will depend on your use case. Do you need to wait for all the calls to be made to do something with the result? Can it be asynchronous?

If you are looking into threads to distribute the work then the third SO post you mentioned is a good way to do it.

If your use case permits the process to be async, I'd definitely look into a scheduler, as mentioned in the first SO post. I've use DelayedJob for this goal, there are some other alternatives. On a related topic, I usually implement a micro-service that receives those requests and processes them async instead of having DelayedJob in the same app, but is just a matter of preference.

Something REALLY important to have in mind if you go with the async approach is that if you are accessing ActiveRecord records inside a thread you need to explicitly check out the database connection. Rails only handles the check in/out of connections in the main thread. Be really careful on this since it can cause connection leaks really hard to track.

The first answer on this SO post shows how to ensure the db connection to be released.

Hope that helps.

Community
  • 1
  • 1
nbermudezs
  • 2,814
  • 1
  • 20
  • 20