4

I am currently doing coding some NN for huge dataset, for example MNIST dataset (about 700*50000). But when I test it, my code got MemoryError. I have a computer with 12 GB ram, but I think Python or Numpy can't use all of them.

Can I push Python or Numpy to use all remaining available memory in my PC ?

OS : Windows 7 64-bit

Python : Python(x, y) 2.7.60

Thanks

psuedobot
  • 141
  • 3
  • 11
  • probably duplicated : [Limit python vm memory](http://stackoverflow.com/questions/1760025/limit-python-vm-memory) you can also use python bindings for win32 to set the max memory usage of a process [Memory limits](http://grokbase.com/t/python/python-win32/04bc6stmqr/memory-limits) – user555742 Apr 06 '14 at 13:07

1 Answers1

6

I believe that the Python(x, y) distribution of Python is still only a 32-bit build (64-bit support is still on its roadmap), so you are limited to 32 bits of address space even though you are using a 64-bit OS. You will need to install a 64-bit build of Python and numpy binaries to get access to more memory.

Robert Kern
  • 13,118
  • 3
  • 35
  • 32
  • owh ok, what is the maximum memory that I can use now ? – psuedobot Apr 07 '14 at 08:02
  • 1
    Maybe 2 Gb for the entire process. In practice, the maximum size of an array that you can allocate is going to be substantially less because of address space fragmentation. Consider installing the standard [64-bit build of Python](https://www.python.org/download/releases/2.7.6) and installing 64-bit numpy from [Christoph Gohlke](http://www.lfd.uci.edu/~gohlke/pythonlibs/). – Robert Kern Apr 09 '14 at 09:22
  • 1
    If I use Anaconda python distribution for 64 bit Windows, can I fully utilized my 64-bit system ? Or the NumPy still have same limitation ? – psuedobot Apr 11 '14 at 12:24
  • Ok Thanks a lot, I already tried it and it works. By the way, is that anyway to show what variable that I have and how much it takes the memory ? – psuedobot Apr 12 '14 at 08:35
  • Not reliably, no, for reasons that the StackOverflow comment box is much too small to explain. Sorry. – Robert Kern Apr 12 '14 at 18:12
  • Owh ok thanks, I just want to know that if it is possible to do – psuedobot Apr 13 '14 at 06:24