-1

I am using PolynomialCountSketch to expand my feature space on a classification problem along with SGDClassifier. The relavent snippet of the code I am using is as follows:

pcs = PolynomialCountSketch(n_components=140, degree=4, gamma=0.75)

X_train_pcs = pcs.fit_transform(X_train)

The first few times I am able to run the code but after a few repetitions with different hyperparameters (in either/both PolynomialCountSketch or SGDCLassifier), I am facing an error which states:

MemoryError: Unable to allocate 1.16 GiB for an array with shape (554181, 140) and data type complex128

I changed my datatype of X_train to float64 and float32 but the same problem exists. I also tried reducing the n_components value but it still gives a similar error. I think it is something to do with my cache memory which gets full after repeated iterations (but I may be wrong as well).

X_train is a array of size 554181x14 of dtype float64 or complex128 (I can work with either). I know it is large but the whole purpose of PolynomialCountSketch is to manage large datasets (or so I thought!!).

I have a laptop with 20GiB of RAM and a core-i5 intel processor.

Can anybody suggest a solution?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
ayps
  • 109
  • 1
  • 1
  • 3
  • https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type – PV8 Aug 08 '23 at 13:59
  • Does this answer your question? [Unable to allocate array with shape and data type](https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type) – PV8 Aug 08 '23 at 13:59
  • No. I increased the Pagefile size on my system but I still get the same error. – ayps Aug 14 '23 at 08:49

0 Answers0