4

My first StackOverflow message after 6 years of using great experience from this site. Thank you all for all the great help you have offered to me and to others.

This problem, however, baffles me completely and I would like to ask for assistance from the community.

I have a 175,759,360 bytes large raw byte 'F' ordered file that I need to read into the memory to analyze using python/numpy. The goal is to obtain a (613, 640, 224) numpy array of np.uint16 dtypes.

I have already succeeded multiple times in similar situations but in the last project I have encountered a problem, which I am unable to resolve.

I was able to succeed before in two ways (np.memmap)

 np.memmap(rawFileName, dtype=np.uint16, mode='r', offset=0, shape=(613, 640, 224), order='F')

or using normal np.fromfile

with open('filename.raw', "rb") as rawFile:
    rawFile = np.fromfile(rawFile, dtype=np.uint16)
    rawFile = rawFile.reshape((613, 640, 224), order="F")

The solution using np.memmap now returns WinError8 error

OSError: [WinError 8] Not enough memory resources are available to process this command

and the solution with fromfile returns reference count error

*** Reference count error detected:
an attempt was made to deallocate 4 (H) ***

I would like to ask for any help I can receive in resolving this matter. The amount of memory should not be a problem since I have tried the code on both 8GB and 32GB memory machines. I am using Win 10 with Python version 3.6.4 and numpy version 1.14.2

Thank you in advance.

Urban Avsec
  • 41
  • 1
  • Maybe a stupid question..: Do you have the 64 bit or 32bit Python version installed? – max9111 Mar 21 '18 at 21:37
  • @max9111 First of all thank you for your comment. My default python is 32bit but I have tried running the same script with 64bit one as well. Crashes as well, just without error. The weird part is that almost the same code works on the same architecture without any problem... Code above was copied from the working code in the first place. The code is run together with PyQt5.. maybe that has an influence on it. – Urban Avsec Mar 22 '18 at 12:13
  • Weird thing... can you simple try to load it this way data= np.fromfile('filename.raw', dtype=np.uint16, count=-1, sep=''") and what do you exactly get? (Nonsense data -> little/big endian,wrong datatype, not expected count (defect in file)).. – max9111 Mar 22 '18 at 12:21
  • @max9111 Tried your suggestion. Does not change much still following is returned *** Reference count error detected: an attempt was made to deallocate 4 (H) **** Tried on two computers right now same error. When I get home from work will try on two more computers. Thank you for trying to help me. – Urban Avsec Mar 22 '18 at 12:52

0 Answers0