python numpy weighted average with nans

Question

First things first: this is not a duplicate of NumPy: calculate averages with NaNs removed, i'll explain why:

Suppose I have an array

a = array([1,2,3,4])

and I want to average over it with the weights

weights = [4,3,2,1]
output = average(a, weights=weights)
print output
     2.0

ok. So this is pretty straightforward. But now I have something like this:

a = array([1,2,nan,4])

calculating the average with the usual method yields of coursenan. Can I avoid this? In principle I want to ignore the nans, so I'd like to have something like this:

a = array([1,2,4])
weights = [4,3,1]
output = average(a, weights=weights)
print output
     1.75

score 16 · Answer 1 · answered Jan 14 '14 at 12:36

16

Alternatively, you can use a MaskedArray as such:

>>> import numpy as np

>>> a = np.array([1,2,np.nan,4])
>>> weights = np.array([4,3,2,1])
>>> ma = np.ma.MaskedArray(a, mask=np.isnan(a))
>>> np.ma.average(ma, weights=weights)
1.75

answered Jan 14 '14 at 12:36

Nicolas Barbey

6,639
4
28
34

This is the best solution because it can be used with `axis` parameter to evaluate the weighted average over several columns where the NaNs do not have the same indices over each column. – Kpym Jun 09 '20 at 09:03

Ashwini Chaudhary · Accepted Answer · 2023-05-01T18:53:59.227

13

First find out indices where the items are not nan, and then pass the filtered versions of a and weights to numpy.average:

>>> import numpy as np
>>> a = np.array([1, 2, np.nan,4])
>>> weights = np.array([4, 3, 2, 1])
>>> indices = np.where(np.logical_not(np.isnan(a)))[0]
>>> np.average(a[indices], weights=weights[indices])
1.75

As suggested by @mtrw in comments, it would be cleaner to use masked array here instead of index array:

>>> indices = ~np.isnan(a)
>>> np.average(a[indices], weights=weights[indices])
1.75

edited May 01 '23 at 18:53

answered Jan 14 '14 at 12:21

Ashwini Chaudhary

244,495
58
464
504

5

+1, though I think `indices = ~np.isnan(a)` looks nicer (and for huge `a` might be faster). – mtrw Jan 14 '14 at 12:36
@mtrw That certainly looks better, will update my answer. Thanks. – Ashwini Chaudhary Jan 14 '14 at 12:38
Another alternative is to use `np.nan_to_num(arr)` before doing average. This will replace any NaN with 0. – Tirtha R Nov 11 '19 at 00:28
@TirthaR The zeros you'd obtain this way would distort the result. – Pauli Jan 13 '20 at 14:12
A problem here is that the method alters the size of the array, so any operation downstream of this and depending on size would need to be corrected accordingly. In such cases it sounds much better to use the masking method proposed by @Nicolas Barbey below. – Pauli Jan 13 '20 at 14:31
@Pauli, True. `np.nan_to_num(arr)` is useful in situations where one would want to ignore NaNs. – Tirtha R Jan 18 '20 at 15:31
The last sentence is misleading, as the associated code block is _not_ displaying what NumPy calls a "masked array": that would be [the `numpy.ma` module](https://numpy.org/doc/stable/reference/maskedarray.generic.html). – Thrastylon Aug 09 '21 at 14:31

score 3 · Answer 3 · answered May 21 '19 at 10:51

I would offer another solution, which is more scalable to bigger dimensions (eg when doing average over different axis). Attached code works with 2D array, which possibly contains nans, and takes average over axis=0.

a = np.random.randint(5, size=(3,2)) # let's generate some random 2D array

# make weights matrix with zero weights at nan's in a
w_vec = np.arange(1, a.shape[0]+1)
w_vec = w_vec.reshape(-1, 1)
w_mtx = np.repeat(w_vec, a.shape[1], axis=1)
w_mtx *= (~np.isnan(a)) 

# take average as (weighted_elements_sum / weights_sum)
w_a = a * w_mtx
a_sum_vec = np.nansum(w_a, axis=0)                                                         
w_sum_vec = np.nansum(w_mtx, axis=0)
mean_vec = a_sum_vec / w_sum_vec

# mean_vec is vector with weighted nan-averages of array a taken along axis=0

ZaxR · Answer 4 · 2020-02-25T18:53:03.120

Expanding on @Ashwini and @Nicolas' answers, here is a version that can also handle an edge case where all the data values are np.nan, and that is designed to also work with pandas DataFrame without type-related issues:

def calc_wa_ignore_nan(df: pd.DataFrame, measures: List[str],
                       weights: List[Union[float, int]]) -> np.ndarray:
    """ Calculates the weighted average of `measures`' values, ex-nans.

    When nans are present in  `measures`' values,
    the weights are recalculated based only on the weights for non-nan measures.

    Note:
        The calculation used is NOT the same as just ignoring nans.
        For example, if we had data and weights:
            data = [2, 3, np.nan]
            weights = [0.5, 0.2, 0.3]
            calc_wa_ignore_nan approach:
                (2*(0.5/(0.5+0.2))) + (3*(0.2/(0.5+0.2))) == 2.285714285714286
            The ignoring nans approach:
                (2*0.5) + (3*0.2) == 1.6

    Args:
        data: Multiple rows of numeric data values with `measures` as column headers.
        measures: The str names of values to select from `row`.
        weights: The numeric weights associated with `measures`.

    Example:
        >>> df = pd.DataFrame({"meas1": [1, 1],
                               "meas2": [2, 2],
                               "meas3": [3, 3],
                               "meas4": [np.nan, 0],
                               "meas5": [5, 5]})
        >>> measures = ["meas2", "meas3", "meas4"]
        >>> weights = [0.5, 0.2, 0.3]
        >>> calc_wa_ignore_nan(df, measures, weights)
        array([2.28571429, 1.6])

    """
    assert not df.empty, "Nothing to calculate weighted average for: `df` is empty."
    # Need to coerce type to np.float instead of python's float
    # to avoid "ufunc 'isnan' not supported for the input types ..." error
    data = np.array(df[measures].values, dtype=np.float64)

    # Make a 2d array with the same weights for each row
    # cast for safety and better errors
    weights = np.array([weights, ] * data.shape[0], dtype=np.float64)

    mask = np.isnan(data)
    masked_data = np.ma.masked_array(data, mask=mask)
    masked_weights = np.ma.masked_array(weights, mask=mask)

    # np.nanmean doesn't support weights
    weighted_avgs = np.average(masked_data, weights=masked_weights, axis=1)
    # Replace masked elements with np.nan
    # otherwise those elements will be interpretted as 0 when read into a pd.DataFrame
    weighted_avgs = weighted_avgs.filled(np.nan)

    return weighted_avgs

score 0 · Answer 5 · answered May 07 '21 at 01:35

All the solutions above are very good, but has don't handle the cases when there is nan in weights. For doing so, using pandas :

def weighted_average_ignoring_nan(df, col_value, col_weight):
  den = 0
  num = 0
  for index, row in df.iterrows():
    if(~np.isnan(row[col_weight]) & ~np.isnan(row[col_value])):
      den = den + row[col_weight]
      num = num + row[col_weight]*row[col_value]
  return num/den

score 0 · Answer 6 · answered Feb 01 '22 at 07:56

Since you're looking for the mean another idea is to simply replace all the nan values with 0's:

>>>import numpy as np
>>>a = np.array([[ 3.,  2.,  5.], [np.nan,  4., np.nan], [np.nan, np.nan, np.nan]])
>>>w = np.array([[ 1.,  2.,  3.], [np.nan, np.nan, np.nan], [np.nan, np.nan, np.nan]])
>>>a[np.isnan(a)] = 0
>>>w[np.isnan(w)] = 0
>>>np.average(a, weights=w)
3.6666666666666665

This can be used with the axis functionality of the average function but be carful that your weights don't sum up to 0.

python numpy weighted average with nans

6 Answers6

Linked