11

I have implemented a function to construct a distance matrix using the jaccard similarity:

import pandas as pd
entries = [
    {'id':'1', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'2', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'3', 'category1':'0', 'category2': '100', 'category3':'100'},
    {'id':'4', 'category1':'100', 'category2': '100', 'category3':'100'},
    {'id':'5', 'category1':'100', 'category2': '0', 'category3':'100'}
           ]
df = pd.DataFrame(entries)

and the distance matrix with scipy

from scipy.spatial.distance import squareform
from scipy.spatial.distance import pdist, jaccard

res = pdist(df[['category1','category2','category3']], 'jaccard')
squareform(res)
distance = pd.DataFrame(squareform(res), index=df.index, columns= df.index)

The problem is that my result looks like this which seems to be false:

enter image description here

What am i missing? The similarity of 0 and 1 have to be maximum for example and the other values seem wrong too

J-H
  • 1,795
  • 6
  • 22
  • 41
  • it looks ok to me, can you an example of a value you think is wrong, and your reasoning? 0 means they agree on every coordinate, 1/3 mean they agree on all but one, 2/3 means they agree on all but 2 and 1 means they disagree on every coordinate – maxymoo Feb 25 '16 at 22:50

1 Answers1

12

Looking at the docs, the implementation of jaccard in scipy.spatial.distance is jaccard dissimilarity, not similarity. This is the usual way in which distance is computed when using jaccard as a metric. The reason for this is because in order to be a metric, the distance between the identical points must be zero.

In your code, the dissimilarity between 0 and 1 should be minimized, which it is. The other values look correct in the context of dissimilarity as well.

If you want similarity instead of dissimilarity, just subtract the dissimilarity from 1.

res = 1 - pdist(df[['category1','category2','category3']], 'jaccard')
root
  • 32,715
  • 6
  • 74
  • 87