1

I have a huge sparse dataset in a dataframe and have been using df.to_sparse but it will be deprecated soon so wanted to switch to pd.Series(pd.SparseArray()) but not sure how to do that for an entire dataframe?

My final df is 100K rows and 49K columns so need an automated way.

Isra
  • 13
  • 4
  • 2
    From the documenation: `Note There’s no performance or memory penalty to using a Series or DataFrame with sparse values, rather than a SparseSeries or SparseDataFrame. ` https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html#sparse-migration – navneethc Jan 03 '20 at 12:14

1 Answers1

1

You could try something like this :

dtype = {key: pd.SparseDtype(df.dtypes[key].type, fill_value=df[key].value_counts().argmax()) for key in df.dtypes.keys()}

df = df.astype(dtype)

And then check the density with df.sparse.density.

This will create sparse data for each column, taking most frequent value as filling value.

(not sure if it's the best approach though)

Alexandre GAREL
  • 53
  • 6