13

For a Recommender System, I need to compute the cosine similarity between all the columns of a whole Spark DataFrame.

In Pandas I used to do this:

import sklearn.metrics as metrics
import pandas as pd

df= pd.DataFrame(...some dataframe over here :D ...)
metrics.pairwise.cosine_similarity(df.T,df.T)

That generates the Similarity Matrix between the columns (since I used the transposition)

Is there any way to do the same thing in Spark (Python)?

(I need to apply this to a matrix made of tens of millions of rows, and thousands of columns, so that's why I need to do it in Spark)

zero323
  • 322,348
  • 103
  • 959
  • 935
Valerio Storch
  • 301
  • 1
  • 3
  • 11

1 Answers1

13

You can use the built-in columnSimilarities() method on a RowMatrix, that can both calculate the exact cosine similarities, or estimate it using the DIMSUM method, which will be considerably faster for larger datasets. The difference in usage is that for the latter, you'll have to specify a threshold.

Here's a small reproducible example:

from pyspark.mllib.linalg.distributed import RowMatrix
rows = sc.parallelize([(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)])

# Convert to RowMatrix
mat = RowMatrix(rows)

# Calculate exact and approximate similarities
exact = mat.columnSimilarities()
approx = mat.columnSimilarities(0.05)

# Output
exact.entries.collect()
[MatrixEntry(0, 2, 0.991935352214),
 MatrixEntry(1, 2, 0.998441152599),
 MatrixEntry(0, 1, 0.997463284056)]
mtoto
  • 23,919
  • 4
  • 58
  • 71
  • 5
    How can I do over the rows instead of the columns? – Charleslmh Oct 12 '17 at 11:00
  • 1
    @mtoto Do you know how to implement the same in Scala https://stackoverflow.com/questions/47010126/calculate-cosine-similarity-spark-dataframe – Moustafa Mahmoud Oct 30 '17 at 07:53
  • 1
    Can you interpret the results of the matrixEntry? like what is 0 and 2? – Reub Jan 08 '18 at 15:46
  • First 2 elements of MatrixEntry correspond to row index and column index. https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/mllib/linalg/distributed/MatrixEntry.html – cylim Aug 16 '19 at 01:33
  • 1
    @Charleslmh in order to do it over rows, you need to get a transpose of the RowMatrix. – prafi Jul 27 '20 at 11:57