Can we improve performance on lists other than java 8 parallel streams

Question

I have to dump data from somewhere by calling rest API which returns List.

First i have to get some List object from one rest api. Now used parallel stream and gone through each item with forEach.
Now on for each element i have to call some other api to get the data which returns again list and save the same list by calling another rest api.
This is taking around 1 Hour for 6000 records of step 1.

I tried like below:

restApiMethodWhichReturns6000Records
    .parallelStream().forEach(id ->{
       anotherMethodWhichgetsSomeDataAndPostsToOtherRestCall(id);
                       });


public void anotherMethodWhichgetsSomeDataAndPostsToOtherRestCall(String id) {

     sestApiToPostData(url,methodThatGetsListOfData(id));
}

That has nothing to do with the performance of lists. It has to do with the speed of the REST services, or of your network, or both. — JB Nizet, Mar 23 '19 at 09:02
looks like you call other rest api sequentially, not concurrently. — Alexei Kaigorodov, Mar 23 '19 at 13:55
@AlexeiKaigorodov Can you improve above code block. I have used parallel stream above. where did i write wrong? — Pavan, Mar 24 '19 at 16:03
@Pavan please provide the code of methods sestApiToPostData and methodThatGetsListOfData. They must be non-blocking. — Alexei Kaigorodov, Mar 24 '19 at 21:41

HariUserX · Answer 1 · 2019-03-23T09:54:41.800

parallelStream can cause unexpected behavior some times. It uses a common ForkJoinPool. So if you have parallel streams somewhere else in the code, it may have a blocking nature for long running tasks. Even in the same stream if some tasks are time taking, all the worker threads will be blocked.

A good discussion on this stackoverflow. Here you see some tricks to assign task specific ForkJoinPool.

First of all make sure your REST service is non-blocking.

One more thing you can do is to play with pool size by supplying -Djava.util.concurrent.ForkJoinPool.common.parallelism=4 to JVM.

score 1 · Answer 2 · answered Apr 26 '19 at 13:41

IF the API calls are blocking, even when you run them in parallel, you will be able to do just a few calls in parallel.

I would try out a solution using CompletableFuture.

The code would be something like this:

List<CompletableFuture>> apiCallsFutures = restApiMethodWhichReturns6000Records
    .stream()
    .map(id -> CompletableFuture.supplyAsync(() -> getListOfData(id))    // Mapping the get list of data call to a Completable Future
                                 .thenApply(listOfData -> callAPItoPOSTData(url, listOfData))   // when the get list call is complete, the post call can be performed 
    .collect(Collectors.toList());

CompletableFuture[] completableFutures = apiCallsFutures.toArray(new CompletableFuture[apiCallsFutures.size()]); // CompletableFuture.allOf accepts only arrays :(

CompletableFuture<Void> all = CompletableFuture.allOf(completableFutures); // Combine all the futures

all.get(); // perform calls

For more details about CompletableFutures, have a look over: https://www.baeldung.com/java-completablefuture

Can we improve performance on lists other than java 8 parallel streams

2 Answers2