0

I'm trying to load a rather big numpy array (~75k images) but i keep getting memory error since the whole dataset is being loaded into memory. I want to use numpy mmap but I keep getting ACCESS_READ error. When I try to load the numpy array using mmap as descriped in this thread I always get the error:

Traceback (most recent call last):

File "mmap.py", line 29, in training_images = np.load('data_small/training_images.npy', mmap_mode='r', encoding = 'latin1')

File "/usr/local/lib/python2.7/site-packages/numpy/lib/npyio.py", line 416, in load return format.open_memmap(file, mode=mmap_mode)

File "/usr/local/lib/python2.7/site-packages/numpy/lib/format.py", line 792, in open_memmap mode=mode, offset=offset)

File "/usr/local/lib/python2.7/site-packages/numpy/core/memmap.py", line 205, in new import mmap

File "/Users/Bjarnason/Desktop/AVEC/mmap.py", line 29, in training_images = np.load('data_small/training_images.npy', mmap_mode='r', encoding = 'latin1')

File "/usr/local/lib/python2.7/site-packages/numpy/lib/npyio.py", line 416, in load return format.open_memmap(file, mode=mmap_mode) File "/usr/local/lib/python2.7/site-packages/numpy/lib/format.py", line 792, in open_memmap mode=mode, offset=offset)

File "/usr/local/lib/python2.7/site-packages/numpy/core/memmap.py", line 257, in new acc = mmap.ACCESS_READ

AttributeError: 'module' object has no attribute 'ACCESS_READ'

Here is the code I'm trying to run:

import numpy as np training_images = np.load('data_small/training_images.npy', mmap_mode='r')

Community
  • 1
  • 1
Aron Björn Bjarnason
  • 1
  • 2

1 Answers1

1

Don't call your file mmap.py. That will shadow the Python mmap library, which is used by numpy.

The file numpy/core/memmap.py in the numpy library executes import mmap. mmap is part of the Python standard library, but your file is also called mmap.py, so the import performed by numpy finds your file instead.

Warren Weckesser
  • 110,654
  • 19
  • 194
  • 214