Running scala futures somewhat in parallel

Question

I have some scala futures. I can easily run them in parallel with Future.sequence. I can also run them one-after-another with something like this:

def serFut[A, B](l: Iterable[A])(fn: A ⇒ Future[B]) : Future[List[B]] =
  l.foldLeft(Future(List.empty[B])) {
    (previousFuture, next) ⇒
      for {
        previousResults ← previousFuture
        next ← fn(next)
      } yield previousResults :+ next
  }

(Described here). Now suppose that I want to run them slightly in parallel - ie with the constraint that at most m of them are running at once. The above code does this for the special case of m=1. Is there a nice scala-idiomatic way of doing it for general m? Then for extra utility, what's the most elegant way to implement a kill-switch into the routine? And could I change m on the fly?

My own solutions to this keep leading me back to procedural code, which feels rather wimpy next to scala elegance.

score 3 · Answer 1 · edited May 23 '17 at 12:29

3

You can achieve it by using an ExecutionContext which uses a pool of max m threads: How to configure a fine tuned thread pool for futures?

Put the implicit val ec = new ExecutionContext { ... somewhere within the scope of the serFut function so that it is used when creating futures.

edited May 23 '17 at 12:29

Community

1
1

answered Jun 10 '15 at 04:09

Rado Buransky

3,252
18
25

Right, `ExecutionContext` would certainly be simple, but do you really mean to use it on `serFut`? That's only built to run one future at a time. Don't you mean to use it on `Future.sequence` ? Also, with `ExecutionContext`, is it ok to throttle `m` up and down while the futures are already running? The way the JVM uses thread pools is invisible to me so I can't really see what's going on. I'd been hoping for a scala function that I can understand and modify as needed. – thund Jun 10 '15 at 14:37
The important point about Futures is that you should stop thinking about threads. It is an abstraction which has certain properties. The only thing you should care about while writing code is that a future will be eventually executed and you can chain other futures with it. If you really need low level control of threads, then go ahead and create threads. – Rado Buransky Jun 11 '15 at 05:32

score 3 · Answer 2 · answered Jun 10 '15 at 05:59

3

The easiest way is to define your ExecutionContext like

implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(m))

answered Jun 10 '15 at 05:59

Dmitry Meshkov

921
7
20

Running scala futures somewhat in parallel

2 Answers2