Scala Future or other way to execute Java void methods in parallel

Question

I have the following piece of code:

stringList map copyURLToFile // List[String]

def copyURLToFile(symbol: String) = {
  val url = new URL(base.replace("XXX", symbol))
  val file = new File(prefix + symbol + ".csv")
  org.apache.commons.io.FileUtils.copyURLToFile(url, file)
}

How do I make copyURLToFile execute in parallel?

Scala Futures need a return type, but if I make copyURLToFile of type Future[Unit], what and how do I return from the function?

possible duplicate of [Parallel map operations?](http://stackoverflow.com/questions/20092855/parallel-map-operations) — senia, Nov 22 '13 at 11:15
You could just add `future` between `=` and `{`like this: `def copyURLToFile(symbol: String) = future{` — senia, Nov 22 '13 at 11:17
I tried having function body in Future {} and recursively going through the list, but it didn't work. Recursion worked but calls to org.apache.commons.io.FileUtils.copyURLToFile didn't (no files were created). I guess the problem was that Java method returns void. — krl, Nov 22 '13 at 13:21
Actually the issue was probably that I was executing that as a standalone program whose main thread terminated before worker threads could execute as described here http://stackoverflow.com/questions/10565475/possible-bug-in-scala-2-10-futures-do-not-run?rq=1 — krl, Nov 22 '13 at 22:31
Void is not a problem per se, the problem is that your main thread does not access the result, thus it doesn't wait for it. — Blaisorblade, Mar 09 '14 at 12:14

score 3 · Accepted Answer · answered Nov 22 '13 at 11:15

3

If you want a very easy way to parallelize this, then just map over the list using the parallel collections library.

stringList.par.map(copyURLToFile).toList

answered Nov 22 '13 at 11:15

Akos Krivachy

4,915
21
28

Works great. Much faster now. – krl Nov 22 '13 at 13:17

score 0 · Answer 2 · answered Nov 22 '13 at 11:16

0

You need to return something, how are you going to know which files where downloaded?

Change your code to be:

def copyURLToFile(symbol: String): (String,String) = {
  val url = new URL(base.replace("XXX", symbol))
  val file = new File(prefix + symbol + ".csv")
  org.apache.commons.io.FileUtils.copyURLToFile(url, file)
  return (symbol, file)
}

And then the resulting collection will contain the symbols an the paths to the file where they were downloaded.

answered Nov 22 '13 at 11:16

Maurício Linhares

39,901
14
121
158

The bigger problem is that the code does not wait for the download to complete, because it does not try to access the results. – Blaisorblade Mar 09 '14 at 12:10

Scala Future or other way to execute Java void methods in parallel

2 Answers2