Scala making parallel network calls using Futures

Question

i'm new to Scala, i have a method, that reads data from the given list of files and does api calls with the data, and writes the response to a file.

listOfFiles.map { file =>
  val bufferedSource = Source.fromFile(file)
  val data = bufferedSource.mkString
  bufferedSource.close()
  val response = doApiCall(data)  // time consuming task
  if (response.nonEmpty) writeFile(response, outputLocation)
}

the above method, is taking too much time, during the network call, so tried to do using parallel processing to reduce the time.

so i tried wrapping the block of code, which consumes more time, but the program ends quickly and its not generating any output, as the above code.

import scala.concurrent.ExecutionContext.Implicits.global

listOfFiles.map { file =>
  val bufferedSource = Source.fromFile(file)
  val data = bufferedSource.mkString
  bufferedSource.close()
  Future {
    val response = doApiCall(data) // time consuming task
    if (response.nonEmpty) writeFile(response, outputLocation)
  }
}

it would be helpful, if you have any suggestions. (I also tried using "par", it works fine, I'm exploring other options other than 'par' and using frameworks like 'akka', 'cats' etc)

it returns Either[String, String], and i use pattern matching and extract the data — chqdrian, May 14 '20 at 10:29
Your main-thread exits and since the global ExecutionContexts' Threads are daemonic, your process exits when main exits. — Viktor Klang, May 14 '20 at 17:44

score 2 · Accepted Answer · answered May 14 '20 at 10:23

Based on Jatin instead of using default execution context which contains deamon threads

import scala.concurrent.ExecutionContext.Implicits.global

define execution context with non-deamon threads

implicit val nonDeamonEc = ExecutionContext.fromExecutor(Executors.newCachedThreadPool)

Also you can use Future.traverse and Await like so

val resultF = Future.traverse(listOfFiles) { file =>
  val bufferedSource = Source.fromFile(file)
  val data = bufferedSource.mkString
  bufferedSource.close()
  Future {
    val response = doApiCall(data) // time consuming task
    if (response.nonEmpty) writeFile(response, outputLocation)
  }
}

Await.result(resultF, Duration.Inf)

traverse converts List[Future[A]] to Future[List[A]].

Scala making parallel network calls using Futures

1 Answers1