What is the most efficient way to remove any matrix from a list of matrices when the matrix contains at least one nan?

Question

I have a list of matrices, M_list and I want to efficiently remove any matrix from the list if it contains a nan

My approach has been to look at each matrix, sum its entries, and remove the matrix from the list if the resulting sum is nan:

import numpy as np
# generate list of seven, 5-by-5 matrices each of which 
contains two nans:

M_list=[]
for t in range(7):
    M=np.random.rand(5,5)
    M.ravel()[np.random.choice(M.size, 2, replace=False)] = 
    np.nan
    M_list.append(M)

# Now my attempt
for t in range(len(M_list)):
    array_sum = np.sum(M_list[t])
    if np.isnan(array_sum):
        M_list.remove(M_list[t])

The error message I receive states, The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(). Removing the last line in the loop removes the error, but I don't see why since I'm merely trying to perform a basic list operation.

Also is there an easier way to accomplish my goal without using a loop?

You haven't given us a sample `M_list` to work with. I guess we are supposed to generate that ourselves? If you post a running script then we can base answers off of that running script. — tdelaney, Aug 31 '21 at 15:22
For a list of seven, five-by-five matrices, each containing two nans, this works: ```for t in range(7): M=np.random.rand(5,5) M.ravel()[np.random.choice(M.size, 2, replace=False)] = np.nan M_list.append(M)``` — fishbacp, Aug 31 '21 at 15:29
Here are a number of ways to check for nan in the individual arrays https://stackoverflow.com/questions/6736590/fast-check-for-nan-in-numpy — tdelaney, Aug 31 '21 at 15:29
Thanks for the code. Add that into the question (along with `import numpy as np`) so a quick copy/paste gives us a running script. — tdelaney, Aug 31 '21 at 15:31

score 1 · Accepted Answer · answered Aug 31 '21 at 15:26

1

You can use isnan and any:

>>> filtered = [arr for arr in M_list if not np.isnan(arr).any()]

answered Aug 31 '21 at 15:26

Sayandip Dutta

15,602
4
23
52

Very elegant. Thanks! – fishbacp Aug 31 '21 at 15:38

score 1 · Answer 2 · answered Aug 31 '21 at 16:10

While the list comprehension is the best method, you should try to understand what's wrong with your attempt.

Make M_list:

In [733]: len(M_list)
Out[733]: 7

we can remove the first list element:

In [734]: M_list.remove(M_list[0])
In [735]: len(M_list)
Out[735]: 6

but if we try to remove the 2nd from the remaining list:

In [736]: M_list.remove(M_list[1])
Traceback (most recent call last):
  File "<ipython-input-736-49c29fd4cfcc>", line 1, in <module>
    M_list.remove(M_list[1])
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This ambiguity error comes up often on SO. It's the result of trying to use a boolean array in a Python context that only accepts a scalar boolean, a simple True/False.

remove has to find the element of the list that "matches" the value. Apparently it iterates through the list until it finds that match. It's been pointed out in other SO, it first tests if the ID is the same. If that fails, it tests for some sort of value equality (==).

That's why removing M_list[0] works, the id's match. But to remove M_list[1] it has to first test that against the first element of the list, resulting in the ambiguity error. We get the same ambiguity error if we try to remove a copy(), because the ID test no longer works.

In [742]: M_list.remove(M_list[0].copy())
Traceback (most recent call last):
  File "<ipython-input-742-6d529bc24c83>", line 1, in <module>
    M_list.remove(M_list[0].copy())
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

But there's another problem with your iterative removal:

for t in range(len(M_list)):
    array_sum = np.sum(M_list[t])
    if np.isnan(array_sum):
        M_list.remove(M_list[t])

M_list.remove() removes an item from the list; the list is shorter, so M_list[t] no longer points to the t element of the original list. Iterative removal like this only works if you iterate from the end. In general for loops on a list only work if the list is not changed (add or remove) in the loop. Otherwise you should be making a new list, leaving the original intact.

What is the most efficient way to remove any matrix from a list of matrices when the matrix contains at least one nan?

2 Answers2