2

I have a large number of data points that I need to perform the same calculations on, roughly 900,000. Each data point is independent of the next. I'm looking for the fastest way to do all the calculations for each. Some crude testing I did compare similar operations using arrays with for loops compared to streams with a map on an ArrayList was not very conclusive, but it appeared that the for loop was faster.

My dev box is very different from the deployed environment as well, so I am not even sure doing it one way on my dev box will result in similar gains in the deployed environment.

I am limited to Java 8.

An example of what I want to do is: Series of two data points obj.a and obj.b Add constant to obj.a + obj.b

With array and for loop

double[] dp1 = ...;//some source
double[] dp2 = ...;//some source
double[] result=new double[dp1.length];
for (int i =0;i < dp1.length;i++){
result[i]=dp1[i]+dp2[i]+myconst;
}

With stream

ArrayList<MyObj> source = ...// some source of obj with dp1 and dp2

ArrayList<Double> result = source.stream().parallel().mapToDouble(a->a.dp1+a.dp2+myconst);

In general, I understand there is overhead with ArrayList which may lead to a minimum size of the Array before seeing gains. Also, if there is a better way to do the stream operations or maybe iterate. Not sure if forEach works in this case as it is a terminal operation and I need to capture the results IN ORIGINAL ORDER of the array/ArrayList.

Dhanushka Rukshan
  • 417
  • 5
  • 15
MJoy99
  • 31
  • 3
  • At a minimum, you should be using `collect` (and likely storing your values in something other than `ArrayList` explicitly). – chrylis -cautiouslyoptimistic- Nov 13 '21 at 04:31
  • 2
    If `dp1[i]+dp2[i]+myconst` is really your operation, then just stick with the sequential `for` loop. There's nothing to gain in parallelizing this. The operation is just so fast. It's only for very, very large-sized arrays that you'll start seeing the benefit of parallelizing it; and 900k isn't even close. – ernest_k Nov 13 '21 at 04:46
  • You could consider OpenCL or jCUDA. https://stackoverflow.com/questions/22866901/using-java-with-nvidia-gpus-cuda – cup Nov 13 '21 at 07:34
  • Also, to add to @ernest_k - A simple for loop will likely execute inside CPU pipeline and within L2/L1 caches, so it should execute most efficiently. Don't burn out your CPU testing :) – K.Nicholas Nov 13 '21 at 17:57
  • The benchmark testing I tried had the array and loop being much faster than ArrayList iteration, ArrayList loop, streams sequential, and streams parallel. Quite a bit of overhead in the non array approaches. – MJoy99 Nov 15 '21 at 00:29

1 Answers1

0

You can consider using parallel-stream if your calculations are independent. But you need to check the time for sure...

At the size 900k, the performance might be not different so much with simple calculation. But it will enhance speed with many heavy calculation for each item.

https://www.baeldung.com/java-when-to-use-parallel-stream

Huy Nguyen
  • 1,931
  • 1
  • 11
  • 11