KneighborsClassifier giving different euclidean value than linalg.norm and scipy.spatial.distance.euclidean

Question

I am trying to implement a knearest neighbors classifier on the mnist dataset.
I tried to check my results by comparing with the Scipy KNeighborsClassifier

For verification I am using the first 6 samples in the training set and finding the 6 nearest neighbors of the first sample in the training set.
The distance that I calculate does not match with the distance given by the KNeighborsClassifier library.
I am not able to figure out why are my values different.

I have referred to this question for getting the euclidean distance.

My code:

from mlxtend.data import loadlocal_mnist
import numpy as np
from scipy.spatial import distance

train, train_label = loadlocal_mnist(
        images_path='train-images.idx3-ubyte', 
        labels_path='train-labels.idx1-ubyte')
train_label = train_label.reshape(-1, 1)

train = train[:6, :]
train_label = train_label[:6, :]
# print(train_label)

test = train.copy()
test_label = train_label.copy()

test = test[:1, :]
test_label = test_label[:1, :]

for test_idx, test_row in enumerate(test):
    for train_idx, train_row in enumerate(train):
        d1 = np.linalg.norm(train_row - test_row)
        d2 = distance.euclidean(train_row, test_row)
        d3 = (((train_row - test_row)**2).sum())**0.5
        d4 = np.dot(train_row - test_row, train_row - test_row)**0.5
        print(train_idx, d1, d2, d3, d4)

Test set is only the first row of train set

The output for the above is:

0 0.0 0.0 0.0 0.0
1 2618.6771469579826 2618.6771469579826 140.3923074815711 15.937377450509228
2 2372.0210791643485 2372.0210791643485 134.29817571359635 10.770329614269007
3 2139.966354875702 2139.966354875702 122.37646832622684 11.313708498984761
4 2485.1432554281455 2485.1432554281455 135.5322839769182 13.892443989449804
5 2582.292392429641 2582.292392429641 144.69968901141425 14.212670403551895

And this is the KNeighborsClassifier code i compare with:

neigh = KNeighborsClassifier(n_neighbors=6)
neigh.fit(train, train_label)
closest = neigh.kneighbors(test[0].reshape(1, -1))
print(closest)

Output:

(array([[   0.        , 2387.11164381, 2554.81975881, 2582.29239243,
        2672.46721215, 2773.14911247]]), array([[0, 1, 3, 5, 4, 2]], dtype=int64))

I am trying to calculate the euclidean distance between the data points to find the nearest neighbors. d1, d2, d3, d4 are 4 different approaches I found from the question linked above and the output are their specific values.
But the distance value I get from the KNeighborsClassifier is different from all of these which also uses euclidean distance as given in the documentation. Why is that happening?

Please make your question *reproducible* (should not be that hard with MNIST); what is `train` & `test`, and how exactly they are built? — desertnaut, Mar 07 '19 at 17:12
Good. What do `d3` & `d4` have to do with the question? They seem irrelevant... — desertnaut, Mar 07 '19 at 21:46

desertnaut · Answer 1 · 2019-03-07T23:50:51.503

OK, here is a hint (have no time currently to look it further, and it may be possibly helpful):

There is certainly something very wrong in the first way you compute the distances (possibly in the way you are slicing the initial data); to see this, let's modify your loops to:

for test_idx, test_row in enumerate(test):
    for train_idx, train_row in enumerate(train):
        d1 = np.linalg.norm(train_row - test_row)
        d2 = np.linalg.norm(test_row - train_row)
        d3 = distance.euclidean(train_row, test_row)
        d4 = distance.euclidean(test_row, train_row)
        print(train_idx, d1, d2, d3, d4)

Here, clearly we should have d1 = d2 = d3 = d4; but the results are:

0 0.0 0.0 0.0 0.0
1 2618.6771469579826 2213.268623552053 2618.6771469579826 2213.268623552053
2 2372.0210791643485 2547.0901044132693 2372.0210791643485 2547.0901044132693
3 2139.966354875702 2374.7201940439213 2139.966354875702 2374.7201940439213
4 2485.1432554281455 2467.6727903026367 2485.1432554281455 2467.6727903026367
5 2582.292392429641 2449.1912951013032 2582.292392429641 2449.1912951013032

i.e. it is d1 = d3 and d2 = d4, but these two quantities are different between them; this should certainly not happen, as the distance is a symmetric function and the order of arguments should play no role:

a = np.array((1, 2, 3))
b = np.array((4, 5, 6))
distance.euclidean(a, b)
# 5.196152422706632
distance.euclidean(b, a)
# 5.196152422706632
np.linalg.norm(a-b)
# 5.196152422706632
np.linalg.norm(b-a)
# 5.196152422706632

Food for thought - hope it helps...

yes that makes sense but still not able to find the source of this bug — otaku, Mar 08 '19 at 01:04

score 0 · Answer 2 · answered Mar 08 '19 at 01:38

I am not sure what was causing this but converting the data from the np.array to a list and then back to an np.array apparently fixed the issue.

train = np.array(train.tolist())
test = np.array(test.tolist())

Thanks to @desertnaut for giving the idea that the issue could be in the slicing of the data but I still can't say for sure what the cause of the issue was.

KneighborsClassifier giving different euclidean value than linalg.norm and scipy.spatial.distance.euclidean

2 Answers2