34

I have a large set of data in which I need to compare the distances of a set of samples from this array with all the other elements of the array. Below is a very simple example of my data set.

import numpy as np
import scipy.spatial.distance as sd

data = np.array(
    [[ 0.93825827,  0.26701143],
     [ 0.99121108,  0.35582816],
     [ 0.90154837,  0.86254049],
     [ 0.83149103,  0.42222948],
     [ 0.27309625,  0.38925281],
     [ 0.06510739,  0.58445673],
     [ 0.61469637,  0.05420098],
     [ 0.92685408,  0.62715114],
     [ 0.22587817,  0.56819403],
     [ 0.28400409,  0.21112043]]
)


sample_indexes = [1,2,3]

# I'd rather not make this
other_indexes = list(set(range(len(data))) - set(sample_indexes))

sample_data = data[sample_indexes]
other_data = data[other_indexes]

# compare them
dists = sd.cdist(sample_data, other_data)

Is there a way to index a numpy array for indexes that are NOT the sample indexes? In my above example I make a list called other_indexes. I'd rather not have to do this for various reasons (large data set, threading, a very VERY low amount of memory on the system this is running on etc. etc. etc.). Is there a way to do something like..

other_data = data[ indexes not in sample_indexes]

I read that numpy masks can do this but I tried...

other_data = data[~sample_indexes]

And this gives me an error. Do I have to create a mask?

Shaido
  • 27,497
  • 23
  • 70
  • 73
b10hazard
  • 7,399
  • 11
  • 40
  • 53
  • Can `data` be arranged so that the *first `N` rows* form the `sample_data` and the remainder form the `other_data`? If so, you could define `sample_data` and `other_data` using basic slices, which return *views*. This would require very little extra memory since the views share the *same underlying data*. – unutbu Aug 15 '14 at 18:32
  • Also, if you are very memory-constrained, you might consider making the file-based arrays using [np.memmap](http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html). – unutbu Aug 15 '14 at 18:33

4 Answers4

41
mask = np.ones(len(data), np.bool)
mask[sample_indexes] = 0
other_data = data[mask]

not the most elegant for what perhaps should be a single-line statement, but its fairly efficient, and the memory overhead is minimal too.

If memory is your prime concern, np.delete would avoid the creation of the mask, and fancy-indexing creates a copy anyway.

On second thought; np.delete does not modify the existing array, so its pretty much exactly the single line statement you are looking for.

Eelco Hoogendoorn
  • 10,459
  • 1
  • 44
  • 42
  • doesn't np.delete create a new copy of the array with the specified elements deleted? I'd rather not create a new array, I'd rather read from the existing one in place. – b10hazard Aug 15 '14 at 17:30
  • yes, delete creates a copy. if memory is really that tight, have you considered storing your data in a pytables array, and operating on that? see f.I. http://pytables.github.io/usersguide/optimization.html#in-kernel-searches – Eelco Hoogendoorn Aug 15 '14 at 17:57
  • 1
    note; if the number of deletions is truly small, a python loop to swap these elements to the end of the array, and then creating a view of the array, would be a simple and efficient solution – Eelco Hoogendoorn Aug 15 '14 at 17:59
  • Okay, that works for my purposes. You said I can do a python loop to swap elements to the end of an array. Can elements be swapped in-place in a numpy array? How would I do this? Thanks – b10hazard Aug 15 '14 at 19:25
  • 2
    This is precisely what the official documentation recommends: https://docs.scipy.org/doc/numpy/reference/generated/numpy.delete.html – pattivacek Feb 10 '17 at 20:47
12

You may want to try in1d

In [5]:

select = np.in1d(range(data.shape[0]), sample_indexes)
In [6]:

print data[select]
[[ 0.99121108  0.35582816]
 [ 0.90154837  0.86254049]
 [ 0.83149103  0.42222948]]
In [7]:

print data[~select]
[[ 0.93825827  0.26701143]
 [ 0.27309625  0.38925281]
 [ 0.06510739  0.58445673]
 [ 0.61469637  0.05420098]
 [ 0.92685408  0.62715114]
 [ 0.22587817  0.56819403]
 [ 0.28400409  0.21112043]]
CT Zhu
  • 52,648
  • 17
  • 120
  • 133
5

You may also use setdiff1d:

In [11]: data[np.setdiff1d(np.arange(data.shape[0]), sample_indexes)]
Out[11]: 
array([[ 0.93825827,  0.26701143],
       [ 0.27309625,  0.38925281],
       [ 0.06510739,  0.58445673],
       [ 0.61469637,  0.05420098],
       [ 0.92685408,  0.62715114],
       [ 0.22587817,  0.56819403],
       [ 0.28400409,  0.21112043]])
Polor Beer
  • 1,814
  • 1
  • 19
  • 18
-3

I'm not familiar with the specifics on numpy, but here's a general solution. Suppose you have the following list:
a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
You create another list of indices you don't want:
inds = [1, 3, 6].
Now simply do this:
good_data = [x for x in a if x not in inds], resulting in good_data = [0, 2, 4, 5, 7, 8, 9].

dpwilson
  • 997
  • 9
  • 19