Is there a simpler way to remove "duplicate" objects from an array (objects with the same property)?

Question

If, given an array of objects, such as:

ArrayList<Person> people = new ArrayList<>(Arrays.aslist(
new Person("Victoria", 25, "Firefighter"),
new Person("Grace", 27, "Footballer"),
new Person("Samantha", 25, "Stock Broker"),
new Person("Victoria", 23, "Poker Player"),
new Person("Jane", 27, "Footballer"),
new Person("Grace", 25, "Security Guard"));

How can one remove any objects that don't have a unique attributes, whilst leaving only one. This could be as simple as duplicate names, which would leave:

Person("Victoria", 25, "Firefighter"),
Person("Grace", 27, "Footballer"),
Person("Samantha", 25, "Stock Broker"),
Person("Jane", 27, "Footballer")

Or more complex, such as jobs that start with the same letter, and the same age:

Person("Victoria", 25, "Firefighter"),
Person("Grace", 27, "Footballer"),
Person("Samantha", 25, "Stock Broker"),
Person("Victoria", 23, "Poker Player"),

So far, the best I've come up with is:

    int len = people.size();
    for (int i = 0; i < len - 1; i++) {
        for (int j = i + 1; j < len; j++)
            if (function(people.get(i), people.get(j))) {
                people.remove(j);
                j--;
                len--;
            }
    }

With "function" checking if the entries are considered "duplicates"

I was wondering if there's a library that does just this, or if you could somehow put this in a lambda expression

implement equals on Person object, and use stream distinct() ---< Returns a stream consisting of the distinct elements (according to Object.equals(Object)) of this stream. — JCompetence, Nov 24 '21 at 12:38
Would you like to provide a fixed or (maybe) varying function for checking dupliated? — chenzhongpu, Nov 24 '21 at 12:42
Your "duplicates" are not necessarily identical to each other. Does it matter, then, which object among each group of duplicates is retained? Also, is it an essential feature that object equivalence is tested via a method? — John Bollinger, Nov 24 '21 at 13:10

MC Emperor · Accepted Answer · 2022-08-15T10:44:32.437

If you say "remove duplicates", the first thing which comes into my mind, is using a Set. However, Set considers an object as "duplicate" if the set already contains an object which is "equal" to that object, by means of the equals method. Implementing Person::equals to check for a job's first letter is not a good fit.

You want to have another sort of 'equals method' for this use case alone. But we have to use something else, as sacrificing equals for this use case alone should not be done.

The Stream interface contains a distinct() method to check for duplicates, but distinct doesn't take a parameter where you can pass in a sort of Comparator or Predicate to define when a Person is considered "distinct" from another Person.

Fortunately, this excellent StackOverflow answer provides exactly what you need:

public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

Now the next thing you must do, is create a record to collect the appropriate object properties:

record PersonAgeAndJobFilter(int age, char jobFirstLetter) {

    public static PersonAgeAndJobFilter ofPerson(Person p) {
        return new PersonAgeAndJobFilter(p.getAge(), p.getJob().charAt(0));
    }
}

Then stream over the people, using your filter:

people.stream()
    .filter(distinctByKey(PersonAgeAndJobFilter::ofPerson))
    .collect(Collectors.toSet());

Alternatively, it can also be achieved with a combination of groupingBy:

Collection<Person> persons = people.stream()
    .collect(groupingBy(PersonAgeAndJobFilter::ofPerson, collectingAndThen(reducing((a, b) -> a), Optional::get)))
    .values();

score 0 · Answer 2 · edited Aug 15 '22 at 10:48

0

ArrayList<Person> new_people = people.stream()
    .distinct()
    .collect(Collectors.toList());

I hope It will work for you.

edited Aug 15 '22 at 10:48

MC Emperor

22,334
15
80
130

answered Nov 24 '21 at 12:40

Nirali Vachhani

15
1
4

1

Streams are a nice idea. But make sure, that your equals() and hashCode() methods are properly overwritten, so that distinct() is working properly. – seism0saurus Nov 24 '21 at 12:56
Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Nov 24 '21 at 16:50

score 0 · Answer 3 · answered Nov 24 '21 at 12:58

You can use a map to pick distinct values by the attribute. This is a method that can be called with a list of objects and a key mapper that picks the attribute that determines uniqueness:

private static <T> List<T> uniqueBy(List<T> objects, 
         Function<T, Object> keyExtractor) {
    return new ArrayList<>(objects.stream()
            .collect(Collectors.toMap(keyExtractor, 
                                      Function.identity(), 
                                      (a, b) -> a, 
                                      LinkedHashMap::new)).values());
}

That uses LinkedHashMap to preserve order from the source list. The method can be used in this way:

List<Person> uniqueByName = uniqueBy(people, Person::getName); //by name
List<Person> uniqueByAge = uniqueBy(people, Person::getAge); //by age
//etc.

Is there a simpler way to remove "duplicate" objects from an array (objects with the same property)?

3 Answers3