0

Spark LSH approxSimilarityJoin is taking too much time:

val column="name"

val new_df=df.select("id", "name", "duns_number", "country_id") 1.7 million record
val new_df_1= df.select("index", "name", "duns_number", "country_id") 0.7 million record

val n_gram = new NGram()
.setInputCol("_"+column)
.setN(4)
.setOutputCol("n_gram_column")

val n_gram_df = n_gram.transform(new_df)
val n_gram_df_1=n_gram.transform(new_df_1)

val validateEmptyVector = udf({ v: Vector => v.numNonzeros > 0 }, DataTypes.BooleanType)

val vectorModeler: CountVectorizerModel = new CountVectorizer()
.setInputCol("n_gram_column")
.setOutputCol("tokenize")
.setVocabSize(456976)
.setMinDF(1)
.fit(n_gram_df)

val vectorizedProductsDF = vectorModeler.transform(n_gram_df)
.filter(validateEmptyVector(col("tokenize")))
.select(col("id"), col(column), col("tokenize"),col("duns_number"),col("country_id"))

val vectorizedProductsDF_1 = vectorModeler.transform(n_gram_df_1)
.filter(validateEmptyVector(col("tokenize")))
.select(col("tokenize"),col(column),col("duns_number"),col("country_id"),col("index"))

val minLshConfig = new MinHashLSH().setNumHashTables(3)
.setInputCol("tokenize")
.setOutputCol("hash")

val lshModel = minLshConfig.fit(vectorizedProductsDF)
val transform_1=lshModel.transform(vectorizedProductsDF)
val transform_2=lshModel.transform(vectorizedProductsDF_1)

val result=lshModel.approxSimilarityJoin(transform_1,transform_2,0.42).toDF

Last line of code (approxSimilarityJoin) is taking too much time and in Stages last few tasks stuck.

I tried with 13 executors with 4 cores each and

spark.sql.shuffle.partitions=600.

Community
  • 1
  • 1
Chandan Kumar Singh
  • 11
  • 1
  • Go easy on / be careful with partitions. You may have a look at this post here : https://stackoverflow.com/questions/48927221/lsh-spark-stucks-forever-at-approxsimilarityjoin-function – Mehdi LAMRANI Dec 26 '19 at 09:16
  • Ye, just by looking at the specs below, these are too much partitions. For 52 cores you should try 100-150 partitions at most (the 'default' suggestion is 2x the number of cores) – KGS Dec 27 '19 at 15:32

0 Answers0