1

I wanted to perform residual analysis on my test set data but got the error "MemoryError: Unable to allocate 812. GiB for an array with shape (330040, 330040) and data type float64".

col_formula = ' + '.join(X_train.columns)
formule = 'Purchase ~ ' + col_formulae

result = smf.ols(formula= formule, data=df).fit()
result.summary2()



train_pred = result.predict(X_train)
resid = y_test - train_pred

After running the last line of code, I got a memory error "MemoryError: Unable to allocate 812. GiB for an array with shape (330040, 330040) and data type float64".

enter image description here enter image description here

How do I fit this error?

Harry Phantom
  • 33
  • 1
  • 5
  • If you expect help, please provide the full text of the Error Message along with the lines of code mentioned in that message. – Claudio Sep 23 '22 at 03:19
  • The most easy fix is to increase the amount of memory to 1 TeraByte ... (joking) – Claudio Sep 23 '22 at 03:22
  • I just edited my post to include pictures of the error code including the line of code the created the error. – Harry Phantom Sep 23 '22 at 03:37
  • What is the shape of `df`? Please provide a [reproducible example](https://stackoverflow.com/q/20109391/1422451). – Parfait Sep 23 '22 at 03:45
  • The `df shape` is (550068, 9). The test and train set is divided by (30:70) respectfully. providing a reproducible will be challenging. – Harry Phantom Sep 23 '22 at 04:38
  • @Claudio You're right, it's an easy fix. ;-) https://yourdatafitsinram.net/ – AKX Sep 23 '22 at 08:32
  • 1
    y_test - train_pred, Most likely one of them is a series and one is dataframe, i.e. one column array - one row array broadcasts to a square array. Make sure both have same dimension. – Josef Sep 23 '22 at 15:33
  • increasing memory just hides bugs in your code a bit longer. – Josef Sep 23 '22 at 15:33
  • 1
    Thanks @Josef it worked. y_test was DataFrame while train_pred was Series. I changed y_test to Series and got the result I wanted. Thanks again – Harry Phantom Sep 24 '22 at 08:53

0 Answers0