0

I currently have some code where I retrieve data from a bunch of URLs. At the moment, I'm doing this sequentially. In order to speed it up, I want to do this concurrently.

To implement this, I'm using the concurrent-ruby gem. Since I have a lot of files which I'm downloading, I want to limit the number of threads. Also, I don't the return value from each thread, since I'm downloading straight into the destination directory.

My question is for this use case, is a thread pool better than using Promises?

It seems that Promises lie at a higher level of abstraction but are useful when one needs the return value of each concurrent operation. Would a thread pool e.g. Concurrent::ThreadPoolExecutior thus be more suitable?

testing09
  • 73
  • 7
  • The docs contain an example for [throttling concurrency](https://ruby-concurrency.github.io/concurrent-ruby/master/file.promises.out.html#Throttling_concurrency) when using promises. With Rails, I'd probably use [Active Job](https://guides.rubyonrails.org/active_job_basics.html)/[Sidekiq](https://github.com/sidekiq/sidekiq) to process such jobs. – Stefan Jul 04 '23 at 12:07

1 Answers1

0

is a thread pool better than using Promises?

It depends. Your described task is quite common and Promises are quite suitable for it. Also in common there is no need to use an abstraction lower than your needs. Keep calm and use Promise until you encounter a trouble with it.

Vasily Kolesnikov
  • 498
  • 2
  • 9
  • _"until you encounter a trouble with it"_ – the OP wants to limit the number of concurrent requests. Maybe you could outline how this can be achieved with promises. – Stefan Jul 04 '23 at 13:31