Numpy stateing that invalid value while calculating normalized mahalanobis distance

Question

Note:

This is for a homework assignment in my data mining class.

I'm going to put relevant code snippets on this SO post, but you can find my entire program at http://pastebin.com/CzNFbLJ2

The dataset I'm using for this program can be found at http://archive.ics.uci.edu/ml/datasets/Iris

So I'm getting: RuntimeWarning: invalid value encountered in sqrt return np.sqrt(m)

I am attempting to find the average Mahalanobis distance of the given iris dataset (for both raw and normalized datasets). The error is only happening on the normalized version of the dataset which is making me wonder if I have incorrectly understood what normalization means (both in code and mathematically).

I thought that normalization means that each component of a vector is divided by it's vector length (causing the vector to add up to 1). I found this SO question How to normalize a 2-dimensional numpy array in python less verbose? and thought it matched up to my concept of normalization. But now my code is reporting that the Mahalanobis distance over the normalized dataset is NAN

def mahalanobis(data):
   import numpy as np;
   import scipy.spatial.distance;
   avg   = 0
   count = 0

   covar = np.cov(data, rowvar=0);
   invcovar = np.linalg.inv(covar)

   for i in range(len(data)):
      for j in range(i + 1, len(data)):
         if(j == len(data)):
            break
         avg += scipy.spatial.distance.mahalanobis(data[i], data[j], invcovar)
         count += 1
   return avg / count


def normalize(data):
   import numpy as np
   row_sums = data.sum(axis=1)
   norm_data = np.zeros((50, 4))
   for i, (row, row_sum) in enumerate(zip(data, row_sums)):
      norm_data[i,:] = row / row_sum
   return norm_data

vector length is the square root of its dot product. you can modify your normalize function by changing the second line to `row_sums = (data*data).sum(axis=1)` — mtadd, Apr 19 '13 at 05:46

score 4 · Accepted Answer · edited Apr 19 '13 at 13:36

4

Probably too late, but check out page 64-65 in our textbook "Introduction to Data Mining". There's a section called "Normalization or Standardization", which explains the concept of normalized data that Hearne is looking for.

Basically, standardized data set x' = (x - mean(x)) / standardDeviation(x)

Since I see you're using python, here's how to do it using SciPy:

normalizedData = (data - data.mean(axis=0)) / data.std(axis=0, ddof=1)

Source: http://mail.scipy.org/pipermail/numpy-discussion/2011-April/056023.html

edited Apr 19 '13 at 13:36

Jaime

65,696
17
124
159

answered Apr 19 '13 at 10:47

dataLandMine

56
2

The book also says normalization can mean (data - data.mean / (data.max - data.min)). I'll try both. – AlexLordThorsen Apr 19 '13 at 17:36

score 3 · Answer 2 · answered Apr 19 '13 at 07:54

3

You can use pdist() to do the distance calculation without for loop:

from sklearn import datasets
iris = datasets.load_iris()
from scipy.spatial.distance import pdist, squareform
print squareform(pdist(iris.data, 'mahalanobis'))

answered Apr 19 '13 at 07:54

HYRY

94,853
25
187
187

score 2 · Answer 3 · edited May 23 '17 at 10:25

Normalization in this context probably does mean subtracting the mean and scaling so the data has a unit covariance matrix.

However, to scale every vector in your dataset to unit norm use: norm_data=data/np.sqrt(np.sum(data*data,1))[:,None].

You need to divide by the L2 norm of each vector, which means squaring the value of each element, then taking the square root of the sum. Broadcasting allows you to avoid explicitly coding the loop (see the answer to the question you cited: https://stackoverflow.com/a/8904762/1149913).

Numpy stateing that invalid value while calculating normalized mahalanobis distance

Note:

3 Answers3