memory issues for sparse one hot encoded features

Question

I want to create sparse matrix for one hot encoded features from data frame df. But I am getting memory issue for code given below. Shape of sparse_onehot is (450138, 1508)

sp_features = ['id', 'video_id', 'genre']
sparse_onehot = pd.get_dummies(df[sp_features], columns = sp_features)
import scipy
X = scipy.sparse.csr_matrix(sparse_onehot.values)

I get memory error as shown below.

MemoryError: Unable to allocate 647. MiB for an array with shape (1508, 450138) and data type uint8

I have tried scipy.sparse.lil_matrix and get same error as above.

Is there any efficient way of handling this? Thanks in advance

what OS do you have? does this help maybe https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type ? — Alena Volkova, Aug 20 '20 at 08:07

score 1 · Accepted Answer · answered Aug 20 '20 at 08:18

1

Try setting to True the sparse parameter:

sparsebool, default False Whether the dummy-encoded columns should be backed by a SparseArray (True) or a regular NumPy array (False).

sparse_onehot = pd.get_dummies(df[sp_features], columns = sp_features, sparse = True)

This will use a much more memory efficient (but somewhat slower) representation than the default one.

answered Aug 20 '20 at 08:18

Ami Tavory

74,578
11
141
185

1

Thanks it solves the error. Later I am able to convert it into `scipy.sparse.csr.csr_matrix ` – Bharat Sharma Aug 20 '20 at 09:00

memory issues for sparse one hot encoded features

1 Answers1