2

Usually pyspark.ml.feature.IDF returns one outputCol that contains SparseVector. All i need is having N-columns with real number values, where N is a number of features defined in IDF(to use that dataframe in catboost later).

I have tried to convert column to array

def dense_to_array(v):
      new_array = list([float(x) for x in v])
      return new_array

dense_to_array_udf = F.udf(dense_to_array, T.ArrayType(T.FloatType()))

data = data.withColumn('tf_idf_features_array', dense_to_array_udf('tf_idf_features'))

and after that use Pandas to convert to columns

data = data.toPandas()
cols = [f'tf_idf_{i}' for i in range(32)]
data = pd.DataFrame(info['tf_idf_features_array'].values.tolist(), columns=cols)

I don't like that way, because i find it really slow. Is there a way to solve my problem over pyspark without pandas?

Gleb
  • 59
  • 7

0 Answers0