0

Say I have the following array:

import numpy as np

data = np.array([[51001, 121, 1, 121212],
                 [51001, 121, 1, 125451],
                 [51001, 125, 1, 127653]]

I want to remove duplicate rows only by the first 3 elements in a row (first 3 columns).

So the result I will get is:

print data
[[51001, 121, 1, 121212],
 [51001, 125, 1, 127653]]

Doesn't matter which row we keep and which row we delete as long as I get the unique by the first 3 columns

Eran Moshe
  • 3,062
  • 2
  • 22
  • 41
  • Slice the first three cols and use the linked dup Q&As. – Divakar Dec 22 '16 at 07:54
  • I Can slice but I dont know how to maintain the 4th column and I didnt see any answer about how to do it – Eran Moshe Dec 22 '16 at 08:08
  • From this [`answer post`](http://stackoverflow.com/a/31097280/3293881), edit : `sorted_idx = np.lexsort(data[:,:3].T)` and `row_mask = np.append([True],np.any(np.diff(sorted_data[:,:3],axis=0),1))`. – Divakar Dec 22 '16 at 08:16

1 Answers1

2

Here's one way using drop_duplicates in pandas

In [179]: pd.DataFrame(data).drop_duplicates([0, 1, 2]).values
Out[179]:
array([[ 51001,    121,      1, 121212],
       [ 51001,    125,      1, 127653]])
Zero
  • 74,117
  • 18
  • 147
  • 154