program freezes on creating a large numpy array

Question

I am making a Tkinter based project where array size can sometimes get as high as 10^9 or even more(although quite minimal chances of more).
Earlier I used a simple array using loops but it took a lot of time in an array of size of order 10^6 or more, so I decided to switch my approach to NumPy and in most cases, it gave far better results, but at the above-mentioned condition(size>=10^9), the program just freezes(sometimes even the computer also freezes leaving no option other than force restart), unlike the simple looping approach which gave the result even for higher sizes of the list, but however, took a whole lot of time.
I looked this but it involved terminologies like using heap memory, stack size and I know little about these things.

I am not quite used to the stack platform, so any advice would be appreciated.

Update: I am adding the chunk of code where I tried replacing normal list with numpy one. Commented lines are the ones I used earlier with simple list.

    def generate(self):
        # t is number of times we need to generate this list
        for i in range(self.t):
            self.n = randint(self.n_min, self.n_max)  # constraints
            # self.a = [0] * self.n
            self.a = np.random.randint(low=self.a_min, high=self.a_max, size=self.n)
            # for j in range(self.n):
            #     self.a[j] = randint(self.a_min, self.a_max)

and then insert all these values in the output screen of Tkinter GUI,
here 'n' i.e size of NumPy array can take very high values sometimes.

I am on dual boot (win+ubuntu) and the current situation is faced on ubuntu in which I have allocated 500 GB storage and my laptop's RAM is 8 GB.

It would be helpful to see the code that is causing the problem, and to know how much memory in your computer. — snakecharmerb, Jan 12 '21 at 18:08
Also, please review [ask] and the other pages about asking questions in the [help] — snakecharmerb, Jan 12 '21 at 18:10
I've added the piece of code causing the issue. Thanks for adding the additional resources, I'll try to abide by these guidelines. — Dude901, Jan 12 '21 at 18:28

score 1 · Answer 1 · answered Jan 12 '21 at 18:24

1

You ran out of memory most likely, for a 1e9 element float64 array in numpy, that would be 8GB alone. Also if you are naively looping over that array (something like):

for item in big_numpy_arrray: 
   do_something(item)

That is going to take forever. Avoid doing that, and use numpy's vector operations when possible.

answered Jan 12 '21 at 18:24

reptilicus

10,290
6
55
79

Actually, all I need to do is to generate that array and insert it in the output widget, looping over that array surely would take a hefty amount of time. I am still learning NumPy and not familiar with all its methods. I've added my laptop specs if that could help in pin pointing the issue. – Dude901 Jan 12 '21 at 18:32
Yeah with 8GB of RAM you are running out of memory – reptilicus Jan 12 '21 at 21:19
Thanks, so what possible alternate do I have so as to be as fast as numpy in such large arrays. I read about using heap memory but couldn't get it clear on how to do it. – Dude901 Jan 13 '21 at 04:39

program freezes on creating a large numpy array

1 Answers1