How can I merge and then distinct collections with the Stream API?

Question

Let's prefix this by my objects equals implementation is not how I need to filter so distinct itself does not work.

class MyObject {
  String foo;
  MyObject( String foo ) {
    this.foo = foo;
  }
  public String getFoo() { return foo; }
}


Collection<MyObject> listA = Arrays.asList("a", "b", "c").stream().map(MyObject::new)
        .collect(Collectors.toList());

Collection<MyObject> listB = Arrays.asList("b", "d").stream().map(MyObject::new)
        .collect(Collectors.toList());


// magic

How can I merge and deduplicate the lists so that the resulting list should be of MyObjects containing "a", "b", "c", "d"?

Note: This is a simplification of what methods we actually need to deduplicate, which are actually complex DTOs of entities loaded by hibernate, but this example should adequately demonstrate the objective.

Answer 1

Such feature is discussed by JDK developers (see JDK-8072723) and might be included in Java-9 (though not guaranteed). The StreamEx library developed by me already has such feature, so you can use it:

List<MyObject> distinct = StreamEx.of(listA).append(listB)
                                  .distinct(MyObject::getFoo).toList();

The StreamEx class is an enhanced Stream which is completely compatible with JDK Stream, but has many additional operations including distinct(Function) which allows you to specify key extractor for distinct operation. Internally it's pretty similar to the solution proposed by @fge.

You can also consider writing custom collector which will combine getting distinct objects and storing them to list:

public static <T> Collector<T, ?, List<T>> distinctBy(Function<? super T, ?> mapper) {
    return Collector.<T, Map<Object, T>, List<T>> of(LinkedHashMap::new,
        (map, t) -> map.putIfAbsent(mapper.apply(t), t), (m1, m2) -> {
            for(Entry<Object, T> e : m2.entrySet()) {
                m1.putIfAbsent(e.getKey(), e.getValue());
            }
            return m1;
        }, map -> new ArrayList<>(map.values()));
}

This collector intermediately collects the results into Map<Key, Element> where Key is the extracted Key and Element is the corresponding stream element. To make sure that exactly first occurring element will be preserved among all repeating ones, the LinkedHashMap is used. Finally you just need to take the values() of this map and dump them into the list. So now you can write:

List<MyObject> distinct = Stream.concat(listA.stream(), listB.stream())
                                .collect(distinctBy(MyObject::getFoo));

If you don't care whether the resulting collection is list or not, you can even remove the new ArrayList<>() step (just using Map::values as a finisher). Also more simplifications are possible if you don't care about order:

public static <T> Collector<T, ?, Collection<T>> distinctBy(Function<? super T, ?> mapper) {
    return Collector.<T, Map<Object, T>, Collection<T>> of(HashMap::new,
            (map, t) -> map.put(mapper.apply(t), t), 
            (m1, m2) -> { m1.putAll(m2); return m1; }, 
            Map::values);
}

Such collector (preserving the order and returning the List) is also available in StreamEx library.

Answer 2

If .equals() does not work for you then you may want to have a go at using Guava's Equivalence.

Provided that your type is T, you need to implement an Equivalence<T>; once you have this, you need to create a:

Set<Equivalence.Wrapper<T>>

into which you'll gather your values. Then, provided your implementation of Equivalence<T> is some static variable named EQ, adding to this set is as simple as:

coll1.stream().map(EQ::wrap).forEach(set::add);
coll2.stream().map(EQ::wrap).forEach(set::add);

And then to obtain a List<T> from this set, you could:

final Set<T> unwrapped = set.stream().map(Equivalence.Wrapper::get)
    .collect(Collectors.toSet());

But of course, since in your comments you say you can do it with a loop, well... Why not keep using that loop?

If it works, don't fix it...

Answer 3

Collection<MyObject> result = Stream.concat(listA.stream(), listB.stream())
                              .filter(distinct(MyObject::getFoo))
                              .collect(Collectors.toList());

public static <T> Predicate<T> distinct(Function<? super T, Object> keyExtractor) {
        Map<Object, String> seen = new ConcurrentHashMap<>();
        return t -> seen.put(keyExtractor.apply(t), "") == null;
    }

I found this distinct function once in a blog (can't remember the link atm).