Error: Out Of Memory, tensorflow cnn

Question

I am using the getting started example of Tensorflow CNN and updating parameters to my own data but since my model is large (244 * 244 features) I got OutOfMemory error.

I am running the training on Ubuntu 14.04 with 4 CPUs and 16Go of RAM. Is there a way to shrink my data so I don't get this OOM error?
My code looks like this:

# Create the Estimator
  mnist_classifier = tf.estimator.Estimator(
    model_fn=cnn_model_fn, model_dir="path/to/model")

# Load the data
train_input_fn = tf.estimator.inputs.numpy_input_fn(
  x={"x": np.array(training_set.data)},
  y=np.array(training_set.target),
  num_epochs=None,
  batch_size=5,
  shuffle=True)

# Train the model
mnist_classifier.train(
  input_fn=train_input_fn,
  steps=100,
  hooks=[logging_hook])

Try reducing the batch size, but I am afraid that it's a learning parameter so it'll affect the net's precision as well. — Ubdus Samad, Jan 12 '18 at 20:20
Of tensorflow's cnn? here it is: https://www.tensorflow.org/tutorials/deep_cnn — Abdou, Jan 13 '18 at 17:31
It's pretty weird that `200 * 244 * 244` doesn't fit in memory, as it's like 40Mb. Maybe you can provide a link to your code? Also there is an argument you doesn't use for `tf.estimator.inputs.numpy_input_fn`: `queue_capacity`. Try to set it to a low value like `batch_size`. — Olivier Moindrot, Jan 15 '18 at 02:40

DarkCygnus · Answer 1 · 2018-01-12T23:57:28.477

4

Is there a way to shrink my data so I don't get this OOM error?

You can slice your training_set to obtain just a portion of the dataset. Something like:

x={"x": np.array(training_set.data)[:(len(training_set)/2)]},
y=np.array(training_set.target)[:(len(training_set)/2)],

In this example you are getting the first half of your dataset (you can select up to what point of your dataset you want to load).

Edit: Another way you can do this is to obtain a random subset of your training dataset. This you can achieve by masking elements on your dataset array. For example:

import numpy as np
from random import random as rn

#obtain boolean mask to filter out some elements
#here you can define your sample %
r = 0.5 #say filter half the elements
mask = [True if rn() >= r else False for i in range(len(training_set))]

#finally, mask out those elements, 
#the result will have ~r times the original elements
reduced_ds = training_set[mask]

edited Jan 12 '18 at 23:57

answered Jan 12 '18 at 15:03

DarkCygnus

7,420
4
36
59

But with this solution I will get only the half of my dataset! How I will load the other half to the training set? – Abdou Jan 12 '18 at 22:59
As I have understand the second solution will randomly ignore half of my training set! This will significantly affect my training accuracy, doesn't it? I want to shrink the data and load them by small pieces, how can I achieve that? – Abdou Jan 12 '18 at 23:12
@Abdou that is completely different thing. In that case, you want to train by batches (sometimes known as Online Learning). That way you can load a batch with enough data to handle, train, and then load the other. As it is quite different to the question you asked here I suggest you post a different question for such purpose, in which case I would gladly tell you how you can make it. – DarkCygnus Jan 12 '18 at 23:56
Here is the question I am really looking for help: https://stackoverflow.com/questions/48236144/how-to-split-tensorflows-cnn-model-by-column-and-train-it-in-small-batches – Abdou Jan 13 '18 at 01:35
@Abdou good, I'll check it in a while I got spare time. In the meantime, consider accepting or voting this answer, in case it solved *this* question (as it seems). – DarkCygnus Jan 13 '18 at 01:36

Error: Out Of Memory, tensorflow cnn

1 Answers1