6

I've been demo testing a bit with lambda performance using Java8 VS. Java8 public functions.

The case is as the following:

  1. I have a list existing of 10 people (5 male and 5 female).

  2. I'd like to know which woman have an age between 18 and 25

Now, when I'm executing these steps a milion times, the results would be:

Lambda with ForEach took: 395 ms (396 ms using JUnit)

Public functions took: 173 ms (169 ms using JUnit)

Lambda with Collect took: 334 ms (335 ms using JUnit)

Now I didn't expect the execution time of lambda to be twice up to six times longer then regular functions.

So, now I'm pretty much wondering whether I've missed something here.

The source can be found here: pastebin.com/BJBk4Tu6


Follow up:

  1. When expanding the list to 1.000.000 items
  2. and filtering all young adult woman once

The results would be:

Lambda with ForEach took: 59 ms

Public functions took: 15 ms

Lambda with Collect took: 12 ms

However, when I'm trying to filter the same list existing of 1.000.000 people 100 times, the results would be:

Lambda with ForEach took: 227 ms

Public functions took: 134 ms

Lambda with Collect took: 172 ms

So, as a final conclusion: Lambdas are quicker when it comes to filtering larger lists while public function (the old way) are quicker at filtering smaller lists.

Also, public functions are quicker when it comes to filtering any lists multiple times, for whatever purpose you'd require to do that.

Latest code: pastebin.com/LcVhgnYv

Community
  • 1
  • 1
lt.kraken
  • 1,287
  • 3
  • 10
  • 27
  • 1
    It could very well be that lambda's are slower than public functions when used with very few results. Have you tried the same benchmark with a lot more results? (say 100k objects) – Kurt Du Bois Oct 08 '14 at 08:57
  • I don't think it makes a difference here, but in general you need to warm up each candidate first before measuring. So run all three methods 10000 times or so before doing the measurements. – Thilo Oct 08 '14 at 08:58
  • Or use Caliper: https://code.google.com/p/caliper/ – The Archetypal Paul Oct 08 '14 at 09:01
  • @KurtDuBois No, i have not yet tried with that many results, but it would be a nice addition, ill come back to this. – lt.kraken Oct 08 '14 at 09:03
  • 1
    I believe warming up makes more of a difference using streams, as there are more method calls involved and thus more code needs to be compiled. – skiwi Oct 08 '14 at 09:07
  • This question cannot be edited until the code is moved into the question itself (since there is an editor restriction on posts containing links to pastebin.com but not featuring code). Larssy1, would you edit the code into the question? As it stands it is not on-topic, since it is not self-contained. Thanks! – halfer Aug 14 '17 at 18:52

1 Answers1

10

As pointed out in the comments: You can hardly draw any conclusion from such a single, simple and isolated microbenchmark run.

Partially quoting from another (otherwise unrelated) answer:

In order to properly and reliably measure execution times, there exist several options. Apart from a profiler, like VisualVM, there are frameworks like JMH or Caliper, but admittedly, using them may be some effort.

For the simplest form of a very basic, manual Java Microbenchmark you have to consider the following:

  • Run the algorithms multiple times, to give the JIT a chance to kick in
  • Run the algorithms alternatingly and not only one after the other
  • Run the algorithms with increasing input size
  • Somehow save and print the results of the computation, to prevent the computation from being optimized away
  • Consider that timings may be distorted by the garbage collector (GC)

These are only rules of thumb, and there may still be unexpected results (refer to the links above for more details). But with this strategy, you usually obtain a good indication about the performance, and at least can see whether it's likely that there really are significant differences between the algorithms.

Related reading:

I applied these basic steps to your program. Here is an MCVE :

NOTE: The remaining part was updated in response to the follow-up edit of the question)

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import java.util.stream.Collectors;

class Person {
    public static final int MALE = 0;
    public static final int FEMALE = 1;
    private final String name;
    private final int sex;
    private final int age;

    public Person(String name, int sex, int age) {
        this.name = name;
        this.sex = sex;
        this.age = age;
    }

    public int getSex() {
        return sex;
    }

    public int getAge() {
        return age;
    }
}

public class Main {

    public static void main(String[] args) {
        new Main();
    }

    private List<Person> people;

    public Main() {

        for (int size=10; size<=1000000; size*=10) {

            Random r = new Random(0);
            people = new ArrayList<Person>();
            for (int i = 0; i < size; i++) {
                int s = r.nextInt(2);
                int a = 25 + r.nextInt(20);
                people.add(new Person("p" + i, s, a));
            }

            int min = 10000000 / size;
            int max = 10 * min;
            for (int n = min; n <= max; n += min) {
                lambdaMethodUsingForEach(n);
                lambdaMethodUsingCollect(n);
                defaultMethod(n);
            }
        }
    }

    public void lambdaMethodUsingForEach(int n) {
        List<Person> lambdaOutput = new ArrayList<Person>();
        long lambdaStart = System.currentTimeMillis();
        for (int i = 0; i < n; i++) {
            lambdaOutput.addAll(getFemaleYoungAdultsUsingLambdaUsingForEach());
        }
        System.out.printf("List size: %10d, runs: %10d, result: %10d, ForEach took: " +
            (System.currentTimeMillis() - lambdaStart) + " ms\n",
            people.size(), n, lambdaOutput.size());
    }

    public void lambdaMethodUsingCollect(int n) {
        List<Person> lambdaOutput = new ArrayList<Person>();
        long lambdaStart = System.currentTimeMillis();
        for (int i = 0; i < n; i++) {
            lambdaOutput.addAll(getFemaleYoungAdultsUsingLambdaUsingCollect());
        }
        System.out.printf("List size: %10d, runs: %10d, result: %10d, collect took: " +
            (System.currentTimeMillis() - lambdaStart) + " ms\n",
            people.size(), n, lambdaOutput.size());
    }

    public void defaultMethod(int n) {
        List<Person> defaultOutput = new ArrayList<Person>();
        long defaultStart = System.currentTimeMillis();
        for (int i = 0; i < n; i++) {
            defaultOutput.addAll(getFemaleYoungAdultsUsingFunctions());
        }
        System.out.printf("List size: %10d, runs: %10d, result: %10d, default took: " +
            (System.currentTimeMillis() - defaultStart) + " ms\n",
            people.size(), n, defaultOutput.size());
    }

    public List<Person> getFemaleYoungAdultsUsingLambdaUsingForEach() {
        List<Person> people = new ArrayList<Person>();
        this.people.stream().filter(
                (p) -> p.getSex() == Person.FEMALE &&
                p.getAge() >= 18 &&
                p.getAge() <= 25).forEach(people::add);
        return people;
    }

    public List<Person> getFemaleYoungAdultsUsingLambdaUsingCollect() {
        return this.people.stream().filter(
                (p) -> p.getSex() == Person.FEMALE &&
                p.getAge() >= 18 &&
                p.getAge() <= 25).collect(Collectors.toList());
    }

    public List<Person> getFemaleYoungAdultsUsingFunctions() {
        List<Person> people = new ArrayList<Person>();
        for (Person p : this.people) {
            if (p.getSex() == Person.FEMALE && p.getAge() >= 18 && p.getAge() <= 25) {
                people.add(p);
            }
        }
        return people;
    }
}

The output on My Machine® is along the lines of this:

    ...
List size:       10, runs:   10000000, result:   10000000, ForEach took: 1482 ms
List size:       10, runs:   10000000, result:   10000000, collect took: 2014 ms
List size:       10, runs:   10000000, result:   10000000, default took: 1013 ms
...
List size:      100, runs:    1000000, result:    3000000, ForEach took: 664 ms
List size:      100, runs:    1000000, result:    3000000, collect took: 515 ms
List size:      100, runs:    1000000, result:    3000000, default took: 441 ms
...
List size:     1000, runs:     100000, result:    2300000, ForEach took: 778 ms
List size:     1000, runs:     100000, result:    2300000, collect took: 721 ms
List size:     1000, runs:     100000, result:    2300000, default took: 841 ms
...
List size:    10000, runs:      10000, result:    2450000, ForEach took: 970 ms
List size:    10000, runs:      10000, result:    2450000, collect took: 971 ms
List size:    10000, runs:      10000, result:    2450000, default took: 1119 ms
...
List size:   100000, runs:       1000, result:    2536000, ForEach took: 976 ms
List size:   100000, runs:       1000, result:    2536000, collect took: 1057 ms
List size:   100000, runs:       1000, result:    2536000, default took: 1109 ms
...
List size:  1000000, runs:        100, result:    2488600, ForEach took: 1323 ms
List size:  1000000, runs:        100, result:    2488600, collect took: 1305 ms
List size:  1000000, runs:        100, result:    2488600, default took: 1422 ms

You can see that the difference between the ForEach and the default (public methods) approach is vanishing even for smaller lists. For larger lists, there even seems to be a slight advantage for the lambda-based approaches.

To emphasize this again: This is a very simple microbenchmark, and even this does not necessarily tell much about the performance of these approaches in practice. However, it is at least reasonable to assume that the difference between the ForEach and the public methods is not as large as suggested by your initial test. Nevertleless: +1 from me for anybody who runs this in JMH or Caliper and posts some further insights about this.

Community
  • 1
  • 1
Marco13
  • 53,703
  • 9
  • 80
  • 159
  • Very usefull comment and I hope you don't mind me using it for research purposes, I am however wondering whether you'd agree with my follow up posted in my question. – lt.kraken Oct 08 '14 at 09:36
  • 2
    @larssy1 I extended the test a little bit, to cover several list sizes. Although the results do not really match your measurements, they at least **indicate** what you said: For smaller lists, the lambdas seem to be a tad slower, but for larger lists, they seem to be as fast as (or even a tad faster than) the "public methods" approach. But... – Marco13 Oct 08 '14 at 09:58
  • 2
    ... but I can't emphasize this often enough: You should be **careful** to not draw too many conclusions from these results. Microbenchmarks can tell you whether something is really odd, but a *profound* analysis requires more effort. You would have to consider GC timings and JVM parameters. And last but not least: The results may also **heavily** depend on *how many persons match the described condition*: There certainly will be a difference implied by whether the resulting list will contain *all* elements of the input list, or *none* ... – Marco13 Oct 08 '14 at 10:03
  • Thanks for your great answer. I have used your main for loop to include expanding sizes and decreasing runs. At this moment, it is as you said depending on many variables. I'll however include this in my initial draft of the research I'm doing. Thanks again! – lt.kraken Oct 08 '14 at 10:29