12

I'm trying to create a very large numpy array of zeros and then copy values from another array into the large array of zeros. I am using Pycharm and I keep getting: MemoryError even when I try and only create the array. Here is how I've tried to create the array of zeros:

import numpy as np

last_array = np.zeros((211148,211148))

I've tried increasing the memory heap in Pycharm from 750m to 1024m as per this question: https://superuser.com/questions/919204/how-can-i-increase-the-memory-heap-in-pycharm, but that doesn't seem to help.

Let me know if you'd like any further clarification. Thanks!

Community
  • 1
  • 1
Andrew Earl
  • 353
  • 1
  • 5
  • 16
  • 13
    You have created an array that is over 100GB in size, assuming the size of `int` is 4 bytes – smac89 May 13 '16 at 15:22
  • Oh lord, I had no idea. That is terrifying. It is a very sparse array though. Is there any way to create an empty array with values in certain positions, such as: `last_array[211148][9]` but everywhere else would be empty? – Andrew Earl May 13 '16 at 15:28
  • This may be helpful: http://stackoverflow.com/questions/1857780/sparse-assignment-list-in-python – Keozon May 13 '16 at 15:35
  • 4
    Or the [`scipy.sparse`](http://docs.scipy.org/doc/scipy/reference/sparse.html) module... – user2390182 May 13 '16 at 15:37
  • It depends on what you are trying to do, but in this case it will be really impossible to create an array that big unless you have the memory for it. In graph problems, we sacrifice speed by making use of an adjacency list. – smac89 May 13 '16 at 15:37
  • 1
    @Smac89 Assuming he's using 64 bit Python, `np.zeros` will create a float64 array by default, in which case he's looking at about 356 GB. – ali_m May 13 '16 at 19:17

1 Answers1

10

Look into using the sparse array capabilities within scipy:
scipy.sparse documentation.

There are a set of examples and tutorials on the scipy.sparse library here:
Scipy lecture notes: Sparse Matrices in SciPy

This may help you solve your memory issues, as well as make everything run faster.

To create an empty sparse array with values in certain positions as you asked in your comment:

Is there any way to create an empty array with values in certain positions, such as: last_array[211147][9] but everywhere else would be empty?

from scipy.sparse import *
values = [42]
row_ind = [211147]
col_ind = [9] 
last_array = csc_matrix((values, (row_ind, col_ind)), shape=(211148,211148))

print(last_array[211147,9])
amicitas
  • 13,053
  • 5
  • 38
  • 50
  • Thanks a ton! I'm working on how to get a bunch of values from one array into specific locations in the sparse matrix now. – Andrew Earl May 13 '16 at 19:29
  • Here is a follow-up question if you are interested: http://stackoverflow.com/questions/37218550/copying-values-of-a-numpy-array-into-specific-location-of-sparse-matrix – Andrew Earl May 13 '16 at 20:08