0

An image in the Massachussets Road Dataset has dimensions (1500,1500,3) . where image_height = image_width = 1500 , and 3 is for RGB channels.

I used skimage.external.tifffile.imread for reading the images and then treid storing them in a numpy array having dimensions ((number_of_images),1500,1500,3) , which gave me a Memory Error.

For solutions, I looked at the PIL module which aids in reshaping the image to a smaller size but I was wondering whether that would cause me to lose some critical information.

Error Message

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-25-4133a180c798> in <module>()
     200 if __name__ == '__main__':
     201 
-->  202     create_train_data()
     203 
     204     create_test_data()

<ipython-input-25-4133a180c798> in create_train_data()
     41 
     42 
---> 43     imgs = np.ndarray((total, image_rows, image_cols,3), dtype=np.uint8)
     44 
     45 

MemoryError:
rayryeng
  • 102,964
  • 22
  • 184
  • 193
Faraz Gerrard Jamal
  • 238
  • 1
  • 3
  • 14
  • Please post the error – user239457 Mar 21 '18 at 07:16
  • Yep, added hyperlink in post which contains error screenshot. – Faraz Gerrard Jamal Mar 21 '18 at 07:30
  • 1
    Possible duplicate of [Numpy memory error creating huge matrix](https://stackoverflow.com/questions/19085012/numpy-memory-error-creating-huge-matrix) – user239457 Mar 21 '18 at 07:41
  • 1
    1500 x 1500 images, compounded by the fact that it is a colour image for classification and especially if you have a lot of images will definitely make you run out of memory. There is no saving of memory here with sparse matrices because the data is dense. You don't have a choice but to resize the images. It also depends on how complex the images are. Resizing the images you can get away with if the images are fairly textured. You also tagged your post with `conv-neural-network`. These full sized images are definitely impractical for a CNN, computation wise and parameter wise. – rayryeng Mar 21 '18 at 07:42
  • [No screenshots, please!](http://meta.stackoverflow.com/a/285557/3005167) This is more work than copy&pasting the error message and bad for several reasons (linked). – MB-F Mar 21 '18 at 07:43
  • 1
    Alright, added code as well! No more screenshots from now on, unless absolutely necessary. – Faraz Gerrard Jamal Mar 21 '18 at 07:54
  • Perfect, thank you. – MB-F Mar 21 '18 at 07:55
  • @rayryeng Alright, seems like resizing will be the way to go. Thanks! – Faraz Gerrard Jamal Mar 21 '18 at 07:56
  • @FarazGerrardJamal No worries. I'll make my comment a formal answer to close off this question. – rayryeng Mar 21 '18 at 07:56

1 Answers1

0

1500 x 1500 images, compounded by the fact that these are colour images for classification and especially if you have a lot of images will definitely make you run out of memory.

There is no saving of memory here with sparse matrices because the data is dense. You don't have a choice but to resize the images. It also depends on how complex the images are.

Resizing the images you can get away with if the images are fairly textured. Given the choice of dataset you've made, we can certainly get away with this. You also tagged your post with . These full sized images are definitely impractical for a CNN, computation wise and parameter wise. Like what I said earlier, it's best that you resize the images before training (and of course testing).

rayryeng
  • 102,964
  • 22
  • 184
  • 193