Numpy: efficient way to extract range of consecutive numbers

Question

Suppose one has a numpy array of floats where all values are in [0,1] e.g.

arr = np.array([
    [0.1,  0.1,  0.1,  0.4,  0.91, 0.81, 0.84], # channel 1 
    [0.81, 0.79, 0.85, 0.1,  0.2,  0.61, 0.91], # channel 2 
    [0.3,  0.1,  0.24, 0.87, 0.62, 1,    0   ], # channel 3
    #...
])

and that one wants to covert this into a binary array. This can be easily done with a cutoff with:

def binary_mask(arr, cutoff=0.5):
  return (arr > cutoff).astype(int)

m = binary_mask(arr)

# array([[0, 0, 0, 0, 1, 1, 1],
#    [1, 1, 1, 0, 0, 1, 1],
#    [0, 0, 0, 1, 1, 1, 0]])

one can get all the indices of the 1s via

for channel in m:
  print(channel.nonzero())

# (array([4, 5, 6]),)
# (array([0, 1, 2, 5, 6]),)
# (array([3, 4, 5]),)

what would be a performant way to the consecutive runs of numbers

e.g.

[ 
    [[4,6]], 
    [[0,2], [5,6]], 
    [[3,5]]
]

a naive approach might be:

def consecutive_integers(arr):

    # indexes of nonzero
    nz = arr.nonzero()[0]

    # storage of "runs"
    runs = []

    # error handle all zero array
    if not len(nz):
        return [[]]

    # run starts with current value
    run_start = nz[0]
    for i in range(len(nz)-1):

        # if run is broken
        if nz[i]+1 != nz[i+1]:
            # store run
            runs.append([run_start, nz[i]])

        # start next run at next number
        run_start = nz[i+1]

    # last run ends with last value
    runs.append([run_start, nz[-1]])

    return runs



print(m)
for runs in [consecutive_integers(c) for c in m]:
    print(runs)



# [[0 0 0 0 1 1 1]
#  [1 1 1 0 0 1 1]
#  [0 0 0 1 1 1 0]]
# 
# [[4, 6]]
# [[0, 2], [5, 6]]
# [[3, 5]]

How would you want it to represent single numbers not part of a sequence? — A Kruger, Nov 18 '18 at 13:38
@AKruger in this case I store it as `[i, i]` just as I store an all zero array as `[[]]`. Should it be counted as a run? probably not... but it is an easy `filter` operation afterwards to make sure `sub[0] != sub[-1]`. Thoughts? — SumNeuron, Nov 18 '18 at 13:40

score 3 · Accepted Answer · answered Nov 18 '18 at 13:42

You can compare consecutive indicators and then use where or flatnonzero

>>> x
array([[0, 0, 0, 0, 1, 1, 1],
       [1, 1, 1, 0, 0, 1, 1],
       [0, 0, 0, 1, 1, 1, 0]])
>>> 
# find switches 0->1 and 1->0
>>> d = np.empty((np.arange(2) + x.shape), bool)
>>> d[:, 0] = x[:, 0]   # a 1 in the first
>>> d[:, -1] = x[:, -1] # or last column counts as a switch
>>> d[:, 1:-1] = x[:, 1:] != x[:, :-1]
>>> 
# find switch indices (of flattened array)
>>> b = np.flatnonzero(d)
# create helper array of row offsets
>>> o = np.arange(0, d.size, d.shape[1])
# split into rows, subtract row offsets and reshape into start, end pairs
>>> result = [(x-y).reshape(-1, 2) for x, y in zip(np.split(b, b.searchsorted(o[1:])), o)]
>>> 
>>> result
[array([[4, 7]]), array([[0, 3],
       [5, 7]]), array([[3, 6]])]

This uses python convention, i.e. right end excluded. If you want right end included use result = [(x-y).reshape(-1, 2) - np.arange(2) for x, y in zip(np.split(b, b.searchsorted(o[1:])), o)] instead.

@SumNeuron not sure they can be compared because it looks to me as if the linked answer only handles 1D. — Paul Panzer, Nov 20 '18 at 11:44

score 2 · Answer 2 · answered Nov 18 '18 at 13:50

2

I would check out this answer: https://stackoverflow.com/a/7353335/1141389

It uses np.split

answered Nov 18 '18 at 13:50

Wesley Bowman

1,366
16
35

score 0 · Answer 3 · answered Nov 18 '18 at 14:21

Using numpy.where you can get it. I have extended you array to try other cases.

import numpy as np

def binary_mask(arr, cutoff=0.5):
  return (arr > cutoff).astype(int)

arr = np.array([
    [0.1,  0.1,  0.1,  0.4,  0.91, 0.81, 0.84, 0, 0.1], # channel 1 
    [0.81, 0.79, 0.85, 0.1,  0.2,  0.61, 0.91, 0,   1], # channel 2 
    [0.3,  0.1,  0.24, 0.87, 0.62, 1,       0, 1,   1], # channel 3
])

m = binary_mask(arr)
for channel in m:
  c = channel.nonzero()[0]
  a = (np.where(np.diff(c) != 1)[0]).tolist()
  a.insert(0,0)
  b = []
  for i, x in enumerate(a):
    if i == len(a)-1 and i > 0:
      b.append([c[x+1], c[-1]])
    elif i == len(a)-1 and i == 0:
      b.append([c[x], c[-1]])
    elif i == 0:
      b.append([c[x], c[a[i+1]]])
    else:
      b.append([c[x+1], c[a[i+1]]])

  print('c = ', c)
  print('a = ', a)
  print('b = ', b)

This program outputs:

c =  [4 5 6]
a =  [0]
b =  [[4, 6]]
c =  [0 1 2 5 6 8]
a =  [0, 2, 4]
b =  [[0, 2], [5, 6], [8, 8]]
c =  [3 4 5 7 8]
a =  [0, 2]
b =  [[3, 5], [7, 8]]

score 0 · Answer 4 · answered Nov 18 '18 at 15:04

I guess there must be a better way to slice arrays by condition between its elements, but I can not find it out.

This is a solution I find so far to slice by consecutive elements and returning the extremes:

import numpy as np

def slice_consecutives(ary):
  pass
  res=[]
  tmp = [ary[0]]
  for idx, x in np.ndenumerate(ary[1:]):
    if x - ary[idx[0]] > 1 or idx[0] + 2 == len(ary):
      if tmp[0] != tmp[-1]: res.append([tmp[0], tmp[-1]])
      tmp = []
    tmp.append(x)
  if ary[-1] - res[-1][-1] == 1: res[-1][-1] = ary[-1]
  return res

ary = np.array([0, 2, 3, 5, 6, 7, 8, 11, 12, 13, 14])
print(slice_consecutives(ary))
# => [[2, 3], [5, 8], [11, 14]]


arry = np.array([4,5,6])
print(slice_consecutives(arry))
#=> [[4, 6]]

Numpy: efficient way to extract range of consecutive numbers

4 Answers4