algorithm removing duplicate elements in array without auxillay storage

Question

I am working on this famous interview question on removing duplicate elements in array without using auxillary storage and preserving the order;

I have read a bunch of posts; Algorithm: efficient way to remove duplicate integers from an array, Removing Duplicates from an Array using C.

They are either implemented in C (without explanation) or the Java Code provided just fails when there is consecutive duplicates such as [1,1,1,3,3].

I am not quite confident with using C, my background is Java. So I implemented the code myself; it follows like this:

1. use two loops, the outer-loop traverses the array and inner loop checks for duplicates and if present replace it with null.
2. Then I go over the duplicate-replaced-null array and remove null elements and replacing it with the next non-null element.

3. The total run-time I see now is O(n^2)+O(n) ~ O(n^2). Reading the above posts, I understood this is the best we can do, if no sorting and auxiliary storage is allowed. My code is here: I am looking for ways to optimize any further (if there is a possibility) or a better/simplisitc logic;

   public class RemoveDup {
       public static void main (String[] args){
           Integer[]  arr2={3,45,1,2,3,3,3,3,2,1,45,2,10};
               Integer[] res= removeDup(arr2);
                   System.out.println(Arrays.toString(res));
               }
             private static Integer[] removeDup(Integer[] data) {
               int size = data.length;
               int count = 1;
                   for (int i = 0; i < size; i++) {
                       Integer temp = data[i];
                       for (int j = i + 1; j < size && temp != null; j++) {
                           if (data[j] == temp) {
                               data[j] = null;
                           }
                       }
                   }
                   for (int i = 1; i < size; i++) {
                       Integer current = data[i];
                       if (data[i] != null) {
                           data[count++] = current;
                       }
                   }
   
                   return Arrays.copyOf(data, count);
   
            }
   

   }

EDIT 1; Reformatted code from @keshlam throws ArrayIndexOutofBound Exception:

private static int removeDupes(int[] array) {
        System.out.println("method called");
        if(array.length < 2)
          return array.length;

        int outsize=1; // first is always kept

     for (int consider = 1; consider < array.length; ++consider) {

          for(int compare=0;compare<outsize;++compare) {
            if(array[consider]!=array[compare])
                array[outsize++]=array[consider]; // already present; advance to next compare
           else break;
          // if we get here, we know it's new so append it to output
          //array[outsize++]=array[consider]; // could test first, not worth it. 

        }

      }
        System.out.println(Arrays.toString(array));
         // length is last written position plus 1
        return outsize;
    }

Answer 1

OK, here's my answer, which should be O(N*N) worst case. (With smaller constant, since even worstcase I'm testing N against -- on average -- 1/2 N, but this is computer science rather than software engineering and a mere 2X speedup isn't significant. Thanks to @Alexandru for pointing that out.)

1) Split cursor (input and output advanced separately),

2) Each new value only has to be compared to what's already been kept, and compare can stop if a match is found. (The hint keyword was "incremental")

3) First element need not be tested.

4) I'm taking advantage of labelled continue where I could have instead set a flag before breaking and then tested the flag. Comes out to the same thing; this is a bit more elegant.

4.5) I could have tested whether outsize==consider and not copied if that was true. But testing for it would take about as many cycles as doing the possibly-unnecessary copy, and the majority case is that they will not be the same, so it's easier to just let a possibly redundant copy take place.

5) I'm not recopying the data in the key function; I've factored out the copy-for-printing operation to a separate function to make clear that removeDupes does run entirely in the target array plus a few automatic variables on the stack. And I'm not spending time zeroing out the leftover elements at the end of the array; that may be wasted work (as in this case). Though I don't think it would actually change the formal complexity.

import java.util.Arrays;

public class RemoveDupes {

  private static int removeDupes(final int[] array) {
    if(array.length < 2)
      return array.length;

    int outsize=1; // first is always kept

    outerloop: for (int consider = 1; consider < array.length; ++consider) {

      for(int compare=0;compare<outsize;++compare)
        if(array[consider]==array[compare])
          continue outerloop; // already present; advance to next compare

      // if we get here, we know it's new so append it to output
      array[outsize++]=array[consider]; // could test first, not worth it. 
    }

    return outsize; // length is last written position plus 1
  }

  private static void printRemoveDupes(int[] array) {
    int newlength=removeDupes(array);
    System.out.println(Arrays.toString(Arrays.copyOfRange(array, 0, newlength)));
  }

  public static void main(final String[] args) {
    printRemoveDupes(new int[] { 3, 45, 1, 2, 3, 3, 3, 3, 2, 1, 45, 2, 10 });
    printRemoveDupes(new int[] { 2, 2, 3, 3 });
    printRemoveDupes(new int[] { 1, 1, 1, 1, 1, 1, 1, 1 });
  }
}

LATE ADDITION: Since folks expressed confusion about point 4 in my explanation, here's the loop rewritten without labelled continue:

for (int consider = 1; consider < array.length; ++consider) {
  boolean matchfound=false;

  for(int compare=0;compare<outsize;++compare) {
    if(array[consider]==array[compare]) {
      matchfound=true;
      break;
    }

    if(!matchFound) // only add it to the output if not found
      array[outsize++]=array[consider];
}

Hope that helps. Labelled continue is a rarely-used feature of Java, so it isn't too surprising that some folks haven't seen it before. It's useful, but it does make code harder to read; I probably wouldn't use it in anything much more complicated than this simple algorithm.

Answer 2

Here one version which doesn't use additional memory (except for the array it returns) and doesn't sort either.

I believe this is slightly worse than O(n*log n).

Edit: I'm wrong. This is slightly better than O(n^3).

public class Dupes {

    private static int[] removeDupes(final int[] array) {
        int end = array.length - 1;
        for (int i = 0; i <= end; i++) {
            for (int j = i + 1; j <= end; j++) {
                if (array[i] == array[j]) {
                    for (int k = j; k < end; k++) {
                        array[k] = array[k + 1];
                    }
                    end--;
                    j--;
                }
            }
        }

        return Arrays.copyOf(array, end + 1);
    }

    public static void main(final String[] args) {
        System.out.println(Arrays.toString(removeDupes(new int[] { 3, 45, 1, 2, 3, 3, 3, 3, 2, 1, 45, 2, 10 })));
        System.out.println(Arrays.toString(removeDupes(new int[] { 2, 2, 3, 3 })));
        System.out.println(Arrays.toString(removeDupes(new int[] { 1, 1, 1, 1, 1, 1, 1, 1 })));
    }
}

and here's a modified version which doesn't shift all of the elements from after the dupe. Instead it simply switches the dupe with the last, non-matching element. This obviously can't guarantee order.

private static int[] removeDupes(final int[] array) {
    int end = array.length - 1;
    for (int i = 0; i <= end; i++) {
        for (int j = i + 1; j <= end; j++) {
            if (array[i] == array[j]) {
                while (end >= j && array[j] == array[end]) {
                    end--;
                }
                if (end > j) {
                    array[j] = array[end];
                    end--;
                }
            }
        }
    }

    return Arrays.copyOf(array, end + 1);
}

Answer 3

Here you have a worst case of O(n^2) where the return points to the first non unique element. So everything before it is unique. Instead of C++ iterators indices in Java can be used.

std::vecotr<int>::iterator unique(std::vector<int>& aVector){
    auto end = aVector.end();
    auto start = aVector.begin();
    while(start != end){
        auto num = *start; // the element to check against
        auto temp = ++start; // start get incremented here
        while (temp != end){
            if (*temp == num){
                std::swap(temp,end);
                end--;
            }
            else
                temp++; // the temp is in else so that if the swap occurs the algo should still check the swapped element.
        }
    }
return end;
}

Java equivalent code: (the return will be an int which is the index of the first not unique element)

int unique(int[] anArray){
        int end = anArray.length-1;
        int start = 0;
        while(start != end){
            int num = anArry[start]; // the element to check against
            int temp = ++start; // start get incremented here
            while (temp != end){
                if (anArry[temp] == num){
                    swap(temp,end); // swaps the values at index of temp and end
                    end--;
                }
                else
                    temp++; // the temp is in else so that if the swap occurs the algo should still check the swapped element.
            }
        }
    return end;
    }

The slight difference in this algo and yours is in your point 2. Where instead of replacing the current element with null you go with swapping it with the last possibly unique element which on the first swap is the last element of array, on second swap the second last and so on.

You might as well consider looking at the std::unique implementation in C++ which is linear in one less than the distance between first and last: Compares each pair of elements, and possibly performs assignments on some of them., but as it was noted by @keshlam it is used on sorted arrays only. The return value is the same as in my algo. Here is the code directly from the standard library:

template<class _FwdIt, class _Pr> inline
    _FwdIt _Unique(_FwdIt _First, _FwdIt _Last, _Pr _Pred)
    {   // remove each satisfying _Pred with previous
    if (_First != _Last)
        for (_FwdIt _Firstb; (_Firstb = _First), ++_First != _Last; )
            if (_Pred(*_Firstb, *_First))
                {   // copy down
                for (; ++_First != _Last; )
                    if (!_Pred(*_Firstb, *_First))
                        *++_Firstb = _Move(*_First);
                return (++_Firstb);
                }
    return (_Last);
    }

Answer 4

To bring in a bit perspective - one solution in Haskell, it uses lists instead of arrays and returns the reversed order, which can be fixed by applying reverse at the end.

import Data.List (foldl')

removeDup :: (Eq a) => [a] -> [a]
removeDup = foldl' (\acc x-> if x `elem` acc then acc else x:acc) []