I have a numpy array that looks like this
[None, None, None, None, None, 8, 7, 2, None, None , None , None, None,
None, None, 169, 37, 9 ,7 ,23, None , None , 111, 24, 8 7 , 9, 12 , 74, None.......]
I need always first and the last value between the None's, the result should look like this
[8,2,169,23,111,74,...]
Does any one know how i can get these numbers back easily ?
A NumPy array that contains None and integers will be of typeobject anyway.
Seems like it it easier to use a list in the first place:
res = []
for x1, x2 in zip(L[:-1], L[1:]):
if (x1 is not None and x2 is None):
res.append(x1)
elif (x1 is None and x2 is not None):
res.append(x2)
None res is:
[8, 2, 169, 23, 111, 74]
To avoid wrong results when the list does not start or end with None, limit the search into the part between the first and last None:
res = []
start = L.index(None)
end = len(L) - L[::-1].index(None)
for x1, x2 in zip(L[start:end-1], L[start+1:end]):
if (x1 is not None and x2 is None):
res.append(x1)
elif (x1 is None and x2 is not None):
res.append(x2)
If you have a NumPy array with NaN instead of None:
a = np.array([np.nan, np.nan, np.nan, np.nan, np.nan, 8, 7, 2, np.nan,
np.nan , np.nan , np.nan, np.nan, np.nan, np.nan, 169, 37, 9,
7 ,23, np.nan , np.nan , 111, 24, 8, 7 , 9, 12 , 74, np.nan])
You can do this in a vectorized way:
b = a[np.isnan(np.roll(a, 1)) | np.isnan(np.roll(a, -1))]
res = b[~np.isnan(b)]
Now res looks like this:
array([ 8., 2., 169., 23., 111., 74.])
Again, a version with limited search between first and last NaN:
indices = np.arange(len(a))[np.isnan(a)]
short = a[indices[0]:indices[-1]]
b = short[np.isnan(np.roll(short, 1)) | np.isnan(np.roll(short, -1))]
res = b[~np.isnan(b)]
Using the pandas package - thanks to innisfree for mentioning bug if series doesn't start/end with None:
import pandas
x=numpy.array([1,3,4,None, None, None, None, None, 8, 7, 2, None, None,7,8])
z = pandas.Series(numpy.append(numpy.insert(x,0,None),None))
res = z[z.shift(1).isnull() | z.shift(-1).isnull()].dropna()
Stealing a lot from this answer you can do this:
Convert your Nones to nan:
x = [None, None, None, None, None, 8, 7, 2, None, None , None , None, None,
None, None, 169, 37, 9 ,7 ,23, None , None , 111, 24, 8 7 , 9, 12 , 74, None]
x = np.array(x,dtype=np.float)
and then:
x = np.vstack([a[s].take([0,-1]) for s in np.ma.clump_unmasked(np.ma.masked_invalid(x))]).flatten()
This divides your array into arrays that correspond to contiguous groups of non-nan values. Then it gets the first and last elements in these arrays using .take([0,-1]). Then it stacks these arrays into one array and flattens it.
print(repr(x))
array([ 8., 2., 169., 23., 111., 74.])
a = [None, None, None, None, None, 8, 7, 2, None, None , None , None, None,
None, None, 169, 37, 9 ,7 ,23, None , None , 111, 24, 8, 7 , 9, 12 , 74, None]
a.append(None)
[a[e] for e in range(len(a)-1) if a[e]!=None and (a[e-1]==None or a[e+1]==None)]
Output:
[8, 2, 169, 23, 111, 74]
A vectorized way with numpy:
arr = np.asarray(arr)
# find all the nones - need the np.array to work around backwards-compatible misbehavior
is_none = arr == np.array(None)
# find which values are on an edge. Start and end get a free pass for being not none
is_edge = ~is_none
is_edge[1:-1] &= (is_none[2:] | is_none[:-2])
# use the boolean mask as an index
res = arr[is_edge]
is_edge could be calculated more verbosely but perhaps more clearly as:
is_edge = ~is_none & (
np.pad(is_none[1:], (0,1), mode='constant', constant_values=True)
|
np.pad(is_none[:-1], (1,0), mode='constant', constant_values=True)
)
Here how I would do it with regular Python lists. I don't have a specific answer with numpy arrays.
result = []
for ind, n in enumerate(lst):
if n is not None and (lst[ind-1] is None or lst[ind+1] is None):
result.append(n)
Note: This will give incorrect result if index 0 or len - 1(last element) is a number.
You could use list comprehension:
ex = [None, None, None, None, None, 8, 7, 2,
None, None , None , None, None, None, None, 169, 37, 9 ,7 ,23,
None , None , 111, 24, 87 , 9, 12 , 74,
None]
filt = [y for x, y, z in zip(ex, ex[1:], ex[2:])
if y is not None and (x is None or z is None)]
# [8, 2, 169, 23, 111, 74]
This requires no external dependencies; however, making two extra copies for my zipped iterator could be costly if the list is particularly big. There are probably ways to overcome this with e.g. itertools.
Note that if your original list has leading or trailing numbers, the above could fail. Strip them first, e.g.
while ex[0] is not None:
del ex[0]
while ex[-1] is not None:
del ex[-1]
The beauty of zip is that you can do this "triple iteration" kind of thing (loop through the list considering three items at a time):
result = []
for previous, item, next in zip(x[1:], x, np.hstack((None, x))):
if item and None in (previous, next):
result.append(item)
The other answers are also reasonable, this was an attempt at readability/understandability.
It's not exactly numpy and it involves a function from another external package: iteration_utilities.split but if you're operating on a list it will probably be relativly fast:
>>> lst = [None, None, None, None, None, 8, 7, 2, None, None , None , None, None,
None, None, 169, 37, 9 ,7 ,23, None , None , 111, 24, 8, 7 , 9, 12 , 74, None]
>>> from iteration_utilities import Iterable, is_None
>>> from operator import itemgetter
>>> Iterable(lst).split(is_None # split by None
).filter(None # remove empty lists
).map(itemgetter(0, -1) # get the first and last element of each splited part
).flatten( # flatten the result
).as_list() # and convert it to a list
[8, 2, 169, 23, 111, 74]
Note that you can also do (something like) this in pure Python as well:
def first_last_between_None(iterable):
last = None
for item in iterable:
# If we're in the None-part just proceed until we get a not None
if last is None:
if item is None:
continue
else:
# Not None, reset last and yield the current value
last = item
yield item
else:
# If the next item is None we're at the end of the number-part
# yield the last item.
if item is None:
yield last
last = item
if last is not None:
yield last
>>> list(first_last_between_None(lst))
[8, 2, 169, 23, 111, 74]
If you want to discard the first and last values when it didn't start or end with None simply take the appropriate slice:
if lst[0] is not None:
res = res[2:]
if lst[-1] is not None:
res = res[:-2]
