remove duplicates from List

Question

I'm looking for a nice way to remove duplicates from a list.

List<String[]> rowList = new ArrayList();
    rowList.add(new String[]{"1","a", "abc"});
    rowList.add(new String[]{"2","b", "def"});
    rowList.add(new String[]{"3","c", "ghi"});
    rowList.add(new String[]{"4","a", "jkl"});
    rowList.add(new String[]{"5","d", "mno"});
    rowList.add(new String[]{"6","e", "pqr"});
    rowList.add(new String[]{"7","b", "stu"});

From this rwoList, i only want entries: 1,2,3,5 and 6. This means i have only one column of intrest, in this case column 2 (a, b, c,..) This is only an easy example i have to handle hugh amount of tables which have 300 columns and min 300000 rows. Another important point is, that i don't won't loose the orientation within the list.

Note: I receive the data from a csv file.

http://stackoverflow.com/questions/10370750/removing-duplicate-elements-from-a-list — Andy897, Oct 09 '13 at 09:07
Every common problem. Seems like no efforts were made to google it. So down voting. — Andy897, Oct 09 '13 at 09:08
Do you retrieve these data from a DB? Is it legal to have duplicates or does it indicate inconsistency? — Fildor, Oct 09 '13 at 09:10
Not directly, I read the data from a csv file into my application. — HaveANiceDay, Oct 09 '13 at 09:12
Why don't you use a specific class to store the data rather than an array of strings? — Vincent van der Weele, Oct 09 '13 at 09:16
@Andy897: I searched a lot, I your link doesn't work for me! — HaveANiceDay, Oct 09 '13 at 09:19
The direction in which Heuster is pointing is: Make a "Data-Class" that implements `equals` in a way, so that it returns true if your critical column (or columns) is the same. Then use a [Set](http://docs.oracle.com/javase/7/docs/api/java/util/Set.html). — Fildor, Oct 09 '13 at 09:19

score 2 · Accepted Answer · answered Oct 09 '13 at 09:28

2

  List<String[]> rowList = new ArrayList<String[]>();
  rowList.add(new String[]{"1","a", "abc"});
  rowList.add(new String[]{"2","b", "def"});
  rowList.add(new String[]{"3","c", "ghi"});
  rowList.add(new String[]{"4","a", "jkl"});
  rowList.add(new String[]{"5","d", "mno"});
  rowList.add(new String[]{"6","e", "pqr"});
  rowList.add(new String[]{"7","b", "stu"});

  Set<String[]> s = new TreeSet<String[]>(new Comparator<String[]>() {
         @Override
         public int compare(String[] o1, String[] o2) {
                return o1[1].compareTo(o2[1]);
        }
   });

Removing the duplicates by adding to set "s"

   s.addAll(rowList);
   List<Object> res = Arrays.asList(s.toArray());

Printing your result

   for (Object object : res) {
         String[] array = (String[])object;
         System.out.println(array[0]+" "+ array[1] +", "+array[2]);
    }

Output

1 a, abc
2 b, def
3 c, ghi
5 d, mno
6 e, pqr

answered Oct 09 '13 at 09:28

Prabhakaran Ramaswamy

25,706
10
57
64

As I understood the OP, the order shall not be changed. This would sort by critical column, wouldn't it? – Fildor Oct 09 '13 at 09:30
I'd suggest using [LinkedHashSet](http://docs.oracle.com/javase/7/docs/api/java/util/LinkedHashSet.html), because: "Hash table and linked list implementation of the Set interface, with predictable iteration order." – Fildor Oct 09 '13 at 09:34
@Fildor Yes you are correct. In order to use LinkedHashSet we need to have pojo with implementation of Comparator or Comparable. I think with String[] it will be little complicated. – Prabhakaran Ramaswamy Oct 09 '13 at 10:13
Thanks a lot! Your solution is exactly what i'm looking for ;) simple and fast enougth. – HaveANiceDay Oct 09 '13 at 10:16

score 1 · Answer 2 · answered Oct 09 '13 at 09:21

Make a custom method isContain(List<String[]> rowList, String string)

private static boolean isContain(List<String[]> rowList, String secStr) {
    for (String[] strings : rowList) {
       if(strings[1].equals(secStr)){
           return true;
       }   
    }
    return false;
}

Check this method before add item to List to remove duplicate item, like:

 List<String[]> rowList = new ArrayList();
    String[] sts= new String[]{"1", "a", "abc"};
    boolean contain= isContain(rowList,sts[1]);
    if(!contain){
        rowList.add(sts);
    }

remove duplicates from List

2 Answers2