What causes python memory error using fillna()

Question

I am getting the following error

numpy.core._exceptions.MemoryError: Unable to allocate array with shape (1329, 22474) and data type float64

This is my code:

interactions_df = pd.DataFrame(list(DB[USER_COLLECTION].find({})))
interactions_full_df = interactions_df.groupby(['personId', 'contentId'])[
    'eventStrength'].sum().apply(smooth_user_preference).reset_index()
users_items_pivot_matrix_df = interactions_full_df.pivot(index='personId',  # edited: interactions_train_df
                                                         columns='contentId',
                                                         values='eventStrength').fillna(0)
users_items_pivot_matrix = users_items_pivot_matrix_df.as_matrix()
users_ids = list(users_items_pivot_matrix_df.index)

In USER_COLLECTION, I have 30009 documents, and 4 columns: _id, personId(string), eventStrength(double), contentId(int32), I tried to settings to increase ram but it didn't work, still couldn't figure it out.

update: After setting custom virtual memory on windows by maximum memory 80000 MB and initial size 5500 MB, and reboot the system, it still have memory error.

Are you using python 32-bit or 64-bit? What operating system are you on? Are you using any special develeopment environment (e.g. Anaconda)? — rexypoo, Feb 03 '20 at 04:48
how much RAM is there on your system? Duplicate of (https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type), maybe — Banks, Feb 03 '20 at 04:49
Your computer is trying to allocate 238 MB with that operation, not that much. — Joe, Feb 03 '20 at 06:06
After setting custom virtual memory on windows by maximum memory 80000 MB and initial size 5500 MB, and reboot the system, it still have memory error. — shanlee, Feb 03 '20 at 07:56

What causes python memory error using fillna()

0 Answers0