Is using a multiprocessing pool in Java encouraged? I have this Python code that I am converting to Java

Question

Python code - Uses a multiprocessing pool and maps an array of strings. On each item in the array, a partial function is used to call a callback_function:

p = multiprocessing.Pool(1)
p.map(functools.partial(callback_function,
                            arg1 = arg1,
                            arg2 = arg2,
                          ), array_string_items)

I am trying to do something similar in Java. I don't see anything in Java API's that is even remotely as simple as the Python code above.

1) Would using threads instead of a processor pool be more encouraged? 2) If so, how would I go about implementing this?

Thanks

You could use ThreadPools. https://docs.oracle.com/javase/tutorial/essential/concurrency/pools.html — apollosoftware.org, Apr 10 '17 at 19:20
Here is a good example: http://tutorials.jenkov.com/java-concurrency/thread-pools.html — apollosoftware.org, Apr 10 '17 at 19:22
@ApolloSoftware great thanks for the link. However, I would also like to know just a general contextual background about why multi-processing isn't as big of a thing, or as available (so it seems) in Java as it is readily-available in Python? What is the disadvantage/advantage about using multi-threading here as opposed to multi-processing as in my Python example above? — TyRyDurden, Apr 10 '17 at 19:30
"multi-processing as available as in Python", heh! Check out google results for `python multithreading only one core` query. It is a bit of a pain in Python too, been there :) — Mikhail Antonov, Apr 10 '17 at 19:35

score 2 · Answer 1 · answered Apr 10 '17 at 19:29

Java8 streams look almost like your example:

Integer ageSum = persons
.parallelStream()
.reduce(0,
    (sum, p) -> {
        System.out.format("accumulator: sum=%s; person=%s\n", sum, p);
        return sum += p.age;
    },
    (sum1, sum2) -> {
        System.out.format("combiner: sum1=%s; sum2=%s\n", sum1, sum2);
        return sum1 + sum2;
    });

As for custom runnables/callables the solution also seems as simple as in Python to me:

List<Callable<Void>> opsList = new ArrayList<>();
opsList.add(someCallable);
// create list of tasks you need to run in parallel...

ExecutorService execSvc = Executors.newFixedThreadPool(4); // 4 threads. check out newCachedThreadPool too
try {
    execSvc.invokeAll(opsList);
} catch (InterruptedException e) {
    e.printStackTrace();
} finally {
    execSvc.shutdown();
}

hoaz · Accepted Answer · 2017-04-11T20:59:19.117

2

Your Python example looks like this in Java:

stringList.parallelStream()
    .map((str) -> callbackFunction.apply(arg1, arg2, str));

Assuming callbackFunction takes three arguments and you apply constant arg1, arg2 arguments to it to make it partial.

You can define partial function explicitly:

int arg1 = 10;
String arg2 = "foo";
Function<String, String> partialFunction = (str) -> callbackFunction.apply(arg1, arg2, str);

And then stream processing becomes ridiculously simple:

stringList.parallelStream().map(partialFunction);

edited Apr 11 '17 at 20:59

answered Apr 10 '17 at 19:34

hoaz

9,883
4
42
53

You say to apply constant arg1 and arg2 arguments to make it a partial. It seems you are applying 'str' as well. Why is that? – TyRyDurden Apr 11 '17 at 20:31
No, `str` is current element of the list, see `(str)` part before callback – hoaz Apr 11 '17 at 20:39
Yes I see that. That totally makes sense I guess I was just wondering if there is a way to do it like a partial in Python, where you don't pass all three arguments in at the same time, you actually give 'default values' for the partial function, and then pass in 'str' separately. – TyRyDurden Apr 11 '17 at 20:44
1

Of course you can do it, you can define explicit partial function, see http://stackoverflow.com/questions/22577261/partial-function-application-in-java-8 for reference. I will update my answer to be more specific – hoaz Apr 11 '17 at 20:45
Excellent. Thank you. – TyRyDurden Apr 11 '17 at 20:59
What if the callback function takes like 8 arguments? – TyRyDurden Apr 11 '17 at 21:39
1

You either pass all 8 directly or create partial with some arguments substituted with constants, similar to my example above with 3 -> 1 arguments. – hoaz Apr 12 '17 at 13:50

Is using a multiprocessing pool in Java encouraged? I have this Python code that I am converting to Java

2 Answers2