MemoryError in scikit even with sparse matrices

Question

I am running (or trying to) a script to classify documents. The code that is throwing the error is:

X = df['text'].values
Y = np.asarray(df['label'], dtype=np.dtype(int))

text_clf = Pipeline([('vect', HashingVectorizer(ngram_range=(1,3), preprocessor=neg_preprocess, n_features=10000000)),
                     ('tfidf', TfidfTransformer()),
                     ('clf', SGDClassifier(loss='log', n_jobs=-1, penalty='elasticnet'))])

text_clf.fit(X, Y)

To get a sense of what the HashingVectorizer produces:

<375175x10000000 sparse matrix of type '<type 'numpy.float64'>'
    with 56324335 stored elements in Compressed Sparse Row format>

The full error and traceback is:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-15-09ad11dfb82b> in <module>()
  7                      ('clf', SGDClassifier(loss='log', n_jobs=-1, penalty='elasticnet'))])
  8 
----> 9 text_clf.fit(X, Y)
 10 
 11 print datetime.now()-startTime

D:\Users\DB\Anaconda\lib\site-packages\sklearn\pipeline.pyc in fit(self, X, y, **fit_params)
129         """
130         Xt, fit_params = self._pre_transform(X, y, **fit_params)
--> 131         self.steps[-1][-1].fit(Xt, y, **fit_params)
132         return self
133 

D:\Users\DB\Anaconda\lib\site-packages\sklearn\linear_model\stochastic_gradient.pyc in fit(self, X, y, coef_init, intercept_init, class_weight, sample_weight)
517                          coef_init=coef_init, intercept_init=intercept_init,
518                          class_weight=class_weight,
--> 519                          sample_weight=sample_weight)
520 
521 

D:\Users\DB\Anaconda\lib\site-packages\sklearn\linear_model\stochastic_gradient.pyc in _fit(self, X, y, alpha, C, loss, learning_rate, coef_init, intercept_init, class_weight, sample_weight)
416 
417         self._partial_fit(X, y, alpha, C, loss, learning_rate, self.n_iter,
--> 418                           classes, sample_weight, coef_init, intercept_init)
419 
420         # fitting is over, we can now transform coef_ to fortran order

D:\Users\DB\Anaconda\lib\site-packages\sklearn\linear_model\stochastic_gradient.pyc in _partial_fit(self, X, y, alpha, C, loss, learning_rate, n_iter, classes, sample_weight, coef_init, intercept_init)
359         if self.coef_ is None or coef_init is not None:
360             self._allocate_parameter_mem(n_classes, n_features,
--> 361                                          coef_init, intercept_init)
362 
363         self.loss_function = self._get_loss_function(loss)

D:\Users\DB\Anaconda\lib\site-packages\sklearn\linear_model\stochastic_gradient.pyc in _allocate_parameter_mem(self, n_classes, n_features, coef_init, intercept_init)
187             else:
188                 self.coef_ = np.zeros((n_classes, n_features),
--> 189                                       dtype=np.float64, order="C")
190 
191             # allocate intercept_ for multi-class

MemoryError:

The size of the feature vector for the whole training set is pretty significant, but each document is quite short (~200 words) and has a small set of features. I would imagine that a sparse matrix would not have trouble handling the data, but perhaps I am completely wrong? I monitored the resource consumption on my computer and it had plenty of RAM left when it failed.

Is there something in the code that is causing this error? I thought that maybe the TfidfTransformer() might be to blame because it causes statefulness, but I removed it from the pipeline and still had the same error. If it's a problem with the feature vector size, surely there's a way to deal with large amounts of data...

I am using ipython notebook and python 2.7.6 Anaconda distribution. If more information is needed to be helpful, please let me know.

Thanks in advance.

score 4 · Accepted Answer · edited May 23 '17 at 10:33

I don't think it is the vectoriser as the traceback shows it fails on the following line:

self.coef_ = np.zeros((n_classes, n_features), dtype=np.float64, order="C")

This allocates a dense numpy array, which uses a lot of memory. It's shape is (n_classes, n_features), and n_features is the same n_features that you passed in as a parameter to the vectoriser, 10M! How many classes do you have in your dataset?

A quick and easy solution is to reduce the value of n_features. Alternatively, you can try other classifiers that do not convert the input to a dense array. I don't know of the top of my head which of sklearn's classifiers do that though.

PS This question shows how to determine the actual in-memory size of a matrix. You can verify it is not the vectoriser or the tfidf transformer that are failing.

MemoryError in scikit even with sparse matrices

1 Answers1