Counting non-zero elements within each row and within each column of a 2D NumPy array

Question

I have a NumPy matrix that contains mostly non-zero values, but occasionally will contain a zero value. I need to be able to:

1. Count the non-zero values in each row and put that count into a variable that I can use in subsequent operations, perhaps by iterating through row indices and performing the calculations during the iterative process.

2. Count the non-zero values in each column and put that count into a variable that I can use in subsequent operations, perhaps by iterating through column indices and performing the calculations during the iterative process.

For example, one thing I need to do is to sum each row and then divide each row sum by the number of non-zero values in each row, reporting a separate result for each row index. And then I need to sum each column and then divide the column sum by the number of non-zero values in the column, also reporting a separate result for each column index. I need to do other things as well, but they should be easy after I figure out how to do the things that I am listing here.

The code I am working with is below. You can see that I am creating an array of zeros and then populating it from a csv file. Some of the rows will contain values for all the columns, but other rows will still have some zeros remaining in some of the last columns, thus creating the problem described above.

The last five lines of the code below are from another posting on this forum. These last five lines of code return a printed list of row/column indices for the zeros. However, I do not know how to use that resulting information to create the non-zero row counts and non-zero column counts described above.

ANOVAInputMatrixValuesArray=zeros([len(TestIDs),9],float)
j=0
for j in range(0,len(TestIDs)):
    TestID=str(TestIDs[j])
    ReadOrWrite='Read'
    fileName=inputFileName
    directory=GetCurrentDirectory(arguments that return correct directory)
    inputfile=open(directory,'r')
    reader=csv.reader(inputfile)
    m=0
    for row in reader:
        if m<9:
            if row[0]!='TestID':
                ANOVAInputMatrixValuesArray[(j-1),m]=row[2]
                m+=1
    inputfile.close()

IndicesOfZeros = indices(ANOVAInputMatrixValuesArray.shape) 
locs = IndicesOfZeros[:,ANOVAInputMatrixValuesArray == 0]
pts = hsplit(locs, len(locs[0]))
for pt in pts:
    print(', '.join(str(p[0]) for p in pt))

Can anyone help me with this?

Answer 1

import numpy as np

a = np.array([[1, 0, 1],
              [2, 3, 4],
              [0, 0, 7]])

columns = (a != 0).sum(0)
rows    = (a != 0).sum(1)

The variable (a != 0) is an array of the same shape as original a and it contains True for all non-zero elements.

The .sum(x) function sums the elements over the axis x. Sum of True/False elements is the number of True elements.

The variables columns and rows contain the number of non-zero (element != 0) values in each column/row of your original array:

columns = np.array([2, 1, 3])
rows    = np.array([2, 3, 1])

EDIT: The whole code could look like this (with a few simplifications in your original code):

ANOVAInputMatrixValuesArray = zeros([len(TestIDs), 9], float)
for j, TestID in enumerate(TestIDs):
    ReadOrWrite = 'Read'
    fileName = inputFileName
    directory = GetCurrentDirectory(arguments that return correct directory)
    # use directory or filename to get the CSV file?
    with open(directory, 'r') as csvfile:
        ANOVAInputMatrixValuesArray[j,:] = loadtxt(csvfile, comments='TestId', delimiter=';', usecols=(2,))[:9]

nonZeroCols = (ANOVAInputMatrixValuesArray != 0).sum(0)
nonZeroRows = (ANOVAInputMatrixValuesArray != 0).sum(1)

EDIT 2:

To get the mean value of all columns/rows, use the following:

colMean = a.sum(0) / (a != 0).sum(0)
rowMean = a.sum(1) / (a != 0).sum(1)

What do you want to do if there are no non-zero elements in a column/row? Then we can adapt the code to solve such a problem.

Answer 2

A fast way to count nonzero elements per row in a scipy sparse matrix m is:

np.diff(m.tocsr().indptr)

The indptr attribute of a CSR matrix indicates the indices within the data corresponding to the boundaries between rows. So calculating the difference between each entry will provide the number of non-zero elements in each row.

Similarly, for the number of nonzero elements in each column, use:

np.diff(m.tocsc().indptr)

If the data is already in the appropriate form, these will run in O(m.shape[0]) and O(m.shape[1]) respectively, rather than O(m.getnnz()) in Marat and Finn's solutions.

If you need both row and column nozero counts, and, say, m is already a CSR, you might use:

row_nonzeros = np.diff(m.indptr)
col_nonzeros = np.bincount(m.indices)

which is not asymptotically faster than first converting to CSC (which is O(m.getnnz())) to get col_nonzeros, but is faster because of implementation details.

Answer 3

The faster way is to clone your matrix with ones instead of real values. Then just sum up by rows or columns:

X_clone = X.tocsc()
X_clone.data = np.ones( X_clone.data.shape )
NumNonZeroElementsByColumn = X_clone.sum(0)
NumNonZeroElementsByRow = X_clone.sum(1)

That worked 50 times faster for me than Finn Årup Nielsen's solution (1 second against 53)

edit: Perhaps you will need to translate NumNonZeroElementsByColumn into 1-dimensional array by

np.array(NumNonZeroElementsByColumn)[0]

Answer 4

For sparse matrices, use the getnnz() function supported by CSR/CSC matrix.

E.g.

a = scipy.sparse.csr_matrix([[0, 1, 1], [0, 1, 0]])
a.getnnz(axis=0)

array([0, 2, 1])

Answer 5

(a != 0) does not work for sparse matrices (scipy.sparse.lil_matrix) in my present version of scipy.

For sparse matrices I did:

    (i,j) = X.nonzero()
    column_sums = np.zeros(X.shape[1])
    for n in np.asarray(j).ravel():
        column_sums[n] += 1.

I wonder if there is a more elegant way.