How to select subsequent numpy arrays handling potential np.nan values

Question

I have a Series like this:

s = pd.Series({10: np.array([[0.72260683, 0.27739317, 0.        ],
                         [0.7187053 , 0.2812947 , 0.        ],
                         [0.71435467, 0.28564533, 1.        ],
                         [0.3268072 , 0.6731928 , 0.        ],
                         [0.31941951, 0.68058049, 1.        ],
                         [0.31260015, 0.68739985, 0.        ]]), 
           20: np.array([[0.7022099 , 0.2977901 , 0.        ],
                         [0.6983866 , 0.3016134 , 0.        ],
                         [0.69411673, 0.30588327, 1.        ],
                         [0.33857735, 0.66142265, 0.        ],
                         [0.33244109, 0.66755891, 1.        ],
                         [0.32675582, 0.67324418, 0.        ]]), 
           20: np.array([[0.68811957, 0.34188043, 0.        ],
                         [0.68425783, 0.31574217, 0.        ],
                         [0.67994496, 0.32005504, 1.        ],
                         [0.34872593, 0.66127407, 1.        ],
                         [0.34276171, 0.65723829, 1.        ],
                         [0.33722803, 0.66277197, 0.        ]]),
           38: np.array([[0.68811957, 0.31188043, 0.        ],
                         [0.68425783, 0.31574217, 0.        ],
                         [0.67994496, 0.32005504, 1.        ],
                         [0.34872593, 0.65127407, 0.        ],
                         [0.34276171, 0.65723829, 1.        ],
                         [0.33722803, 0.66277197, 0.        ]]),
           np.nan: np.nan}
)

I want to subset it with np.array([1, 4, 1, 5]) or np.array([1, 4, 1, np.nan]) returning np.nan no matter what the value is on the last element of indices array. How can I accomplish that?

Please note that I can't simply remove last element of a Series.

jezrael · Accepted Answer · 2020-04-09T05:54:34.927

You can modify previous answer with remove missing values of Series and last add them by Series.reindex (only necessary unique index of Series):

#a = np.array([1, 4, 1, 5])
a = np.array([1, 4, 1, np.nan])

mask = s.notna()
b = np.array(s[mask].tolist())[np.arange(mask.sum()), a[mask].astype(int), 2]
print (b)
[0. 1. 0.]

c = pd.Series(b, index=s[mask].index).reindex(s.index)
print (c)
10.0    0.0
20.0    1.0
38.0    0.0
NaN     NaN
dtype: float64

EDIT: If not unique values in index is necessary create unique MultiIndex with GroupBy.cumcount:

s = pd.Series({10: np.array([[0.72260683, 0.27739317, 0.        ],
                         [0.7187053 , 0.2812947 , 0.        ],
                         [0.71435467, 0.28564533, 1.        ],
                         [0.3268072 , 0.6731928 , 0.        ],
                         [0.31941951, 0.68058049, 1.        ],
                         [0.31260015, 0.68739985, 0.        ]]), 
           20: np.array([[0.7022099 , 0.2977901 , 0.        ],
                         [0.6983866 , 0.3016134 , 0.        ],
                         [0.69411673, 0.30588327, 1.        ],
                         [0.33857735, 0.66142265, 0.        ],
                         [0.33244109, 0.66755891, 1.        ],
                         [0.32675582, 0.67324418, 0.        ]]), 
           23: np.array([[0.68811957, 0.34188043, 0.        ],
                         [0.68425783, 0.31574217, 0.        ],
                         [0.67994496, 0.32005504, 1.        ],
                         [0.34872593, 0.66127407, 1.        ],
                         [0.34276171, 0.65723829, 1.        ],
                         [0.33722803, 0.66277197, 0.        ]]),
           38: np.array([[0.68811957, 0.31188043, 0.        ],
                         [0.68425783, 0.31574217, 0.        ],
                         [0.67994496, 0.32005504, 1.        ],
                         [0.34872593, 0.65127407, 0.        ],
                         [0.34276171, 0.65723829, 1.        ],
                         [0.33722803, 0.66277197, 0.        ]]),
           np.nan: np.nan}
).rename({23:20})

print (s)
10.0    [[0.72260683, 0.27739317, 0.0], [0.7187053, 0....
20.0    [[0.7022099, 0.2977901, 0.0], [0.6983866, 0.30...
20.0    [[0.68811957, 0.34188043, 0.0], [0.68425783, 0...
38.0    [[0.68811957, 0.31188043, 0.0], [0.68425783, 0...
NaN                                                   NaN
dtype: object

a = np.array([1, 4, 1, 2, np.nan])

s = s.to_frame('a').set_index(s.groupby(s.index).cumcount(), append=True)['a']
print (s)
10.0  0    [[0.72260683, 0.27739317, 0.0], [0.7187053, 0....
20.0  0    [[0.7022099, 0.2977901, 0.0], [0.6983866, 0.30...
      1    [[0.68811957, 0.34188043, 0.0], [0.68425783, 0...
38.0  0    [[0.68811957, 0.31188043, 0.0], [0.68425783, 0...
NaN   0                                                  NaN
Name: a, dtype: object

mask = s.notna()
b = np.array(s[mask].tolist())[np.arange(mask.sum()), a[mask].astype(int), 2]
print (b)
[0. 1. 0. 1.]

c = pd.Series(b, index=s[mask].index).reindex(s.index)
print (c)
10.0  0    0.0
20.0  0    1.0
      1    0.0
38.0  0    1.0
NaN   0    NaN
dtype: float64

And in last step remove helper level of MultiIndex:

c = c.reset_index(level=-1, drop=True)
print (c)
10.0    0.0
20.0    1.0
20.0    0.0
38.0    1.0
NaN     NaN
dtype: float64

It works, thanks a bunch! As a side question, can I use `np.stack(s[mask])` instead of `np.array(s[mask].tolist()`? Both seem to return the same thing, but maybe is there any peculiarity? — jakes, Apr 09 '20 at 06:21

How to select subsequent numpy arrays handling potential np.nan values

1 Answers1