Removing Duplicate Values from ArrayList

Question

I have one Arraylist of String and I have added Some Duplicate Value in that. and i just wanna remove that Duplicate value So how to remove it.

Here Example I got one Idea.

List<String> list = new ArrayList<String>();
        list.add("Krishna");
        list.add("Krishna");
        list.add("Kishan");
        list.add("Krishn");
        list.add("Aryan");
        list.add("Harm");

        System.out.println("List"+list);

        for (int i = 1; i < list.size(); i++) {
            String a1 = list.get(i);
            String a2 = list.get(i-1);
            if (a1.equals(a2)) {
                list.remove(a1);
            }
        }

        System.out.println("List after short"+list);

But is there any Sufficient way remove that Duplicate form list. with out using For loop ? And ya i can do it by using HashSet or some other way but using array list only. would like to have your suggestion for that. thank you for your answer in advance.

Answer 1

You can create a LinkedHashSet from the list. The LinkedHashSet will contain each element only once, and in the same order as the List. Then create a new List from this LinkedHashSet. So effectively, it's a one-liner:

list = new ArrayList<String>(new LinkedHashSet<String>(list))

Any approach that involves List#contains or List#remove will probably decrease the asymptotic running time from O(n) (as in the above example) to O(n^2).

---

EDIT For the requirement mentioned in the comment: If you want to remove duplicate elements, but consider the Strings as equal ignoring the case, then you could do something like this:

Set<String> toRetain = new TreeSet<String>(String.CASE_INSENSITIVE_ORDER);
toRetain.addAll(list);
Set<String> set = new LinkedHashSet<String>(list);
set.retainAll(new LinkedHashSet<String>(toRetain));
list = new ArrayList<String>(set);

It will have a running time of O(n*logn), which is still better than many other options. Note that this looks a little bit more complicated than it might have to be: I assumed that the order of the elements in the list may not be changed. If the order of the elements in the list does not matter, you can simply do

Set<String> set = new TreeSet<String>(String.CASE_INSENSITIVE_ORDER);
set.addAll(list);
list = new ArrayList<String>(set);

Answer 2

if you want to use only arraylist then I am worried there is no better way which will create a huge performance benefit. But by only using arraylist i would check before adding into the list like following

void addToList(String s){
  if(!yourList.contains(s))
       yourList.add(s);
}

In this cases using a Set is suitable.

Answer 3

You can make use of Google Guava utilities, as shown below

 list = ImmutableSet.copyOf(list).asList(); 

This is probably the most efficient way of eliminating the duplicates from the list and interestingly, it preserves the iteration order as well.

UPDATE

But, in case, you don't want to involve Guava then duplicates can be removed as shown below.

ArrayList<String> list = new ArrayList<String>();
    list.add("Krishna");
    list.add("Krishna");
    list.add("Kishan");
    list.add("Krishn");
    list.add("Aryan");
    list.add("Harm");

System.out.println("List"+list);
HashSet hs = new HashSet();
hs.addAll(list);
list.clear();
list.addAll(hs);

But, of course, this will destroys the iteration order of the elements in the ArrayList.

Shishir

Answer 4

Java 8 stream function

You could use the distinct function like above to get the distinct elements of the list,

stringList.stream().distinct();

From the documentation,

Returns a stream consisting of the distinct elements (according to Object.equals(Object)) of this stream.

---

Another way, if you do not wish to use the equals method is by using the collect function like this,

stringList.stream()  
    .collect(Collectors.toCollection(() -> 
        new TreeSet<String>((p1, p2) -> p1.compareTo(p2)) 
));  

From the documentation,

Performs a mutable reduction operation on the elements of this stream using a Collector.

Hope that helps.

Answer 5

Simple function for removing duplicates from list

private void removeDuplicates(List<?> list)
{
    int count = list.size();

    for (int i = 0; i < count; i++) 
    {
        for (int j = i + 1; j < count; j++) 
        {
            if (list.get(i).equals(list.get(j)))
            {
                list.remove(j--);
                count--;
            }
        }
    }
}

Example:
Input: [1, 2, 2, 3, 1, 3, 3, 2, 3, 1, 2, 3, 3, 4, 4, 4, 1]
Output: [1, 2, 3, 4]

Answer 6

List<String> list = new ArrayList<String>();
        list.add("Krishna");
        list.add("Krishna");
        list.add("Kishan");
        list.add("Krishn");
        list.add("Aryan");
        list.add("Harm");

HashSet<String> hs=new HashSet<>(list);

System.out.println("=========With Duplicate Element========");
System.out.println(list);
System.out.println("=========Removed Duplicate Element========");
System.out.println(hs);

Answer 7

I don't think the list = new ArrayList<String>(new LinkedHashSet<String>(list)) is not the best way , since we are using the LinkedHashset(We could use directly LinkedHashset instead of ArrayList),

Solution:

import java.util.ArrayList;
public class Arrays extends ArrayList{

@Override
public boolean add(Object e) {
    if(!contains(e)){
        return super.add(e);
    }else{
        return false;
    }
}

public static void main(String[] args) {
    Arrays element=new Arrays();
    element.add(1);
    element.add(2);
    element.add(2);
    element.add(3);

    System.out.println(element);
}
}

Output: [1, 2, 3]

Here I am extending the ArrayList , as I am using the it with some changes by overriding the add method.

Answer 8

     public List<Contact> removeDuplicates(List<Contact> list) {
    // Set set1 = new LinkedHashSet(list);
    Set set = new TreeSet(new Comparator() {
        @Override
        public int compare(Object o1, Object o2) {
                 if(((Contact)o1).getId().equalsIgnoreCase(((Contact)2).getId()) ) {
                return 0;
            }
            return 1;
        }
    });
    set.addAll(list);
    final List newList = new ArrayList(set);
    return newList;
}

Answer 9

This will be the best way

    List<String> list = new ArrayList<String>();
    list.add("Krishna");
    list.add("Krishna");
    list.add("Kishan");
    list.add("Krishn");
    list.add("Aryan");
    list.add("Harm");

    Set<String> set=new HashSet<>(list);

Answer 10

It is better to use HastSet

1-a) A HashSet holds a set of objects, but in a way that it allows you to easily and quickly determine whether an object is already in the set or not. It does so by internally managing an array and storing the object using an index which is calculated from the hashcode of the object. Take a look here

1-b) HashSet is an unordered collection containing unique elements. It has the standard collection operations Add, Remove, Contains, but since it uses a hash-based implementation, these operation are O(1). (As opposed to List for example, which is O(n) for Contains and Remove.) HashSet also provides standard set operations such as union, intersection, and symmetric difference.Take a look here

2) There are different implementations of Sets. Some make insertion and lookup operations super fast by hashing elements. However that means that the order in which the elements were added is lost. Other implementations preserve the added order at the cost of slower running times.

The HashSet class in C# goes for the first approach, thus not preserving the order of elements. It is much faster than a regular List. Some basic benchmarks showed that HashSet is decently faster when dealing with primary types (int, double, bool, etc.). It is a lot faster when working with class objects. So that point is that HashSet is fast.

The only catch of HashSet is that there is no access by indices. To access elements you can either use an enumerator or use the built-in function to convert the HashSet into a List and iterate through that.Take a look here

Answer 11

Without a loop, No! Since ArrayList is indexed by order rather than by key, you can not found the target element without iterate the whole list.

A good practice of programming is to choose proper data structure to suit your scenario. So if Set suits your scenario the most, the discussion of implementing it with List and trying to find the fastest way of using an improper data structure makes no sense.

Answer 12

public static void main(String[] args) {
    @SuppressWarnings("serial")
    List<Object> lst = new ArrayList<Object>() {
        @Override
        public boolean add(Object e) {
            if(!contains(e))
            return super.add(e);
            else
            return false;
        }
    };
    lst.add("ABC");
    lst.add("ABC");
    lst.add("ABCD");
    lst.add("ABCD");
    lst.add("ABCE");
    System.out.println(lst);

}

This is the better way

Answer 13

list = list.stream().distinct().collect(Collectors.toList());
This could be one of the solutions using Java8 Stream API. Hope this helps.

Answer 14

 public void removeDuplicates() {
    ArrayList<Object> al = new ArrayList<Object>();
    al.add("java");
    al.add('a');
    al.add('b');
    al.add('a');
    al.add("java");
    al.add(10.3);
    al.add('c');
    al.add(14);
    al.add("java");
    al.add(12);

    System.out.println("Before Remove Duplicate elements:" + al);
    for (int i = 0; i < al.size(); i++) {
        for (int j = i + 1; j < al.size(); j++) {
            if (al.get(i).equals(al.get(j))) {
                al.remove(j);
                j--;
            }
        }
    }
    System.out.println("After Removing duplicate elements:" + al);
}

Before Remove Duplicate elements:

[java, a, b, a, java, 10.3, c, 14, java, 12]

After Removing duplicate elements:

[java, a, b, 10.3, c, 14, 12]

Answer 15

Using java 8:

public static <T> List<T> removeDuplicates(List<T> list) {
    return list.stream().collect(Collectors.toSet()).stream().collect(Collectors.toList());
}

Answer 16

In case you just need to remove the duplicates using only ArrayList, no other Collection classes, then:-

//list is the original arraylist containing the duplicates as well
List<String> uniqueList = new ArrayList<String>();
    for(int i=0;i<list.size();i++) {
        if(!uniqueList.contains(list.get(i)))
            uniqueList.add(list.get(i));
    }

Hope this helps!

Answer 17

private static void removeDuplicates(List<Integer> list)
{
    Collections.sort(list);
    int count = list.size();
    for (int i = 0; i < count; i++) 
    {
        if(i+1<count && list.get(i)==list.get(i+1)){
            list.remove(i);
            i--;
            count--;
        }
    }
}

Answer 18

public static List<String> removeDuplicateElements(List<String> array){
    List<String> temp = new ArrayList<String>();
    List<Integer> count = new ArrayList<Integer>();
    for (int i=0; i<array.size()-2; i++){
        for (int j=i+1;j<array.size()-1;j++)
            {
                if (array.get(i).compareTo(array.get(j))==0) {
                    count.add(i);
                    int kk = i;
                }
            }
        }
        for (int i = count.size()+1;i>0;i--) {
            array.remove(i);
        }
        return array;
    }
}