0

I've read that python is able to use the entire physical memory available on the machine, therefore it should not run out of memory before actually filling up all the free 9+GB of my laptop.

However, using laspy to parse a 10M point cloud (200MB) and selecting points in the following way produces an out of memory error:

inFile = File(sys.argv[1], mode = "r")
all_points = np.vstack([inFile.x, inFile.y, inFile.z, inFile.return_num, inFile.intensity]).transpose()
lower_points = all_points[ 1 > inFile.z ]
upper_points = all_points[ 1 <= inFile.z ]

The last conditional selection triggers the memory error. There are actually 2M points that satisfy the first condition, and 10M points in total, so 8M points should satisfy the second condition.

If I change upper_points to be a normal list (as:[] ) and .append every point which z is bigger than 1, it works without problems.

8M points should be approximately 200MB or a bit more, so I don't really understand the problem. What am I missing?

pmod
  • 10,450
  • 1
  • 37
  • 50
Lake
  • 4,072
  • 26
  • 36
  • 1
    You may not have enough *contiguous* memory? Lists can consist of objects all over the place in memory, but numpy arrays' elements need to be adjacent to one another in memory. – acdr May 12 '16 at 09:26
  • Possible, but with 9+GB free and esimated 1.2 GB of usage, I would presume that it should work... weird – Lake May 12 '16 at 11:59
  • 1
    Is your python distribution 32 or 64? – armatita May 12 '16 at 17:13
  • It's 32, but total memory allocation should be less than 2GB... it's 5 floating points (4bytes) times 10 millions, 40MB, add some overhead (10 times?) and its still'400 MB... I don't see how it could get close to 2 GB – Lake May 13 '16 at 10:26
  • @armatita Actually I tried with python 64 bits and it works, I'm still puzzled why that'd be the case as the allocation wouldn't be more than 2GB... – Lake May 13 '16 at 16:33
  • 1
    Besides the limits of 32bits operative systems typically limit the amount of memory a process can use. Also remember numpy arrays are more a complex object than the typical C array (for several reasons) and they do spend more memory. It's not that simple to calculate how much a numpy array is spending. Check for instance this link: http://stackoverflow.com/questions/11784329/python-memory-usage-of-numpy-arrays – armatita May 13 '16 at 17:29
  • Thanks for the tip^^ I'll accept it as an answer if you want^^ – Lake May 13 '16 at 18:46

0 Answers0