0

I have 3 usage GPUs (Nvidia GeForce 12GB each) available on a server. I have a very large trainingset of about 300k 256x256 images. The model I want to train contains about 3M parameters. I get a memory error:

MemoryError: Unable to allocate array with shape (39155, 256, 256) and data type float64

39155*256*256*8 Bytes = 20.5 GB

therefore I (guess?) this cannot be trained on a 12GB card. How (can you provide a piece of code) to use all 3 GPUs?

SheppLogan
  • 322
  • 3
  • 18
  • You are wrong to say it can't be trained on a 12 GB card, this is exactly why we use batches and Keras has generator so you don't have to load all the data at the same time. – Dr. Snoopy Jan 03 '20 at 01:09
  • Well it is obviously a memory problem since I get the MEMORY ERROR message... maybe you havent read my post entirely... but what you are referring to, is is the batch size?(in which case mine is 32) – SheppLogan Jan 04 '20 at 19:39
  • I never said its not a memory problem, its just that we already have solutions for this. Check here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly – Dr. Snoopy Jan 04 '20 at 19:45
  • ok thanks for the link. In fact I think the whole dataset cannot fit RAM (not VRAM) entirely it is about 60GB. At first I thought the whole dataset would be loaded into the VRAM but clearly its impossible. If it is, as it seems, loaded in RAM, I am surprised that the swap partition on disk cannot handle the extra data (at the cost of speed)? – SheppLogan Jan 04 '20 at 22:35
  • The whole point of a generator and batching is that it is not necessary to load the whole dataset into RAM. Only a batch of data is loaded at a given time. – Dr. Snoopy Jan 04 '20 at 22:36
  • Ok, but are we talking about a batch as the batch size we defined on which the gradient is computed or another batch? – SheppLogan Jan 04 '20 at 23:25
  • The batch size defines the number of elements of a batch, as you say where the gradient of the loss is computed. – Dr. Snoopy Jan 04 '20 at 23:42
  • ah maybe my mistake was to think that given a batch size I defined e.g. 32, only 32*256*256*8 Bytes of data would be loaded, i.e. you say that it is exactly this process that I must code, otherwise the whole dataset e.g. 300k would be loaded in memory (or would try to).. – SheppLogan Jan 05 '20 at 13:34
  • Could you solve your problem? I have the same of yours. Have you check out this question https://stackoverflow.com/questions/45132940/numpy-memmap-memory-usage-want-to-iterate-once? – Marlon Teixeira Aug 25 '20 at 23:31

0 Answers0