1

recently I had to use a list of a large number of lists for a program. I tried to make a large list of lists, by running list= [[] for _ in range(200000000)] on the command line, and I opened Task Manager. Firstly, Python started using up more and more RAM, until it ate up 10GB out of my 11.9GB of RAM. Then, I saw a dip in the usage of RAM on python to about 3000MB, and the Task Manager showed that python started using Disk. Does that mean that python 64-bit starts saving variables in the hard drive when it doesn't have enough RAM left?

If so, is there a way to do it manually, and how fast is it? (a bit of a relative question, I expect an answer like "it takes 30 times more to lookup from SSD than from RAM", and, "Python can transfer 30MB/s from RAM to a Hard Drive, or store variables in a Hard Drive without using RAM").

P.S. I got an SSD.

Lainad
  • 199
  • 9
  • 2
    You're seeing swap memory being used when physical memory is depleted. – AKX Jul 03 '20 at 19:37
  • @AKX Is there a way to store a variable somewhere that is not in the physical memory straight away? – Lainad Jul 03 '20 at 19:40
  • 1
    You’re seeing your _operating system_ make more virtual memory available as demand from python increases - as soon as demand exceeds available RAM then your operating system moves some current physical RAM onto disk. Python doesn’t know this is happening, it just asks your OS for more RAM and the OS tries to be helpful. SSDs might be 1000000x slower than physical RAM if you’re lucky. You don’t want swapping to happen because it kills your application’s performance, so you should redesign your code to not use so much memory. You can selectively store stuf to disk, you’ll have to code this. – DisappointedByUnaccountableMod Jul 03 '20 at 19:47
  • @barny is the pickle module used for that? What do you suggest? – Lainad Jul 03 '20 at 19:52
  • You could pickle (some) objects, or save data to yaml or json. Better to save intermediate stages of data processing than normal runtime variables - it has to be big so save a lot of ram. – DisappointedByUnaccountableMod Jul 03 '20 at 20:09
  • Check out this thread. It might answer some questions you have: https://stackoverflow.com/questions/5537618/memory-errors-and-list-limits/5537764 – Nathan Silverman Jul 03 '20 at 21:52

0 Answers0