How to load only batch data to in memory during training with estimator?

Question

currently I train my model with estimator using below code. But when I use large dataset, my in memory (RAM) is not sufficient to load large dataset. So, is there a way to load only batch data to in memory while training with estimator?

here example shown for keras. How can I implement it using estimator?

currently I am loading all my data in memory and feeding it to estimator.

classifier = tf.estimator.Estimator(model_fn = convNet,model_dir='/dir')
train_input_fn = tf.estimator.inputs.numpy_input_fn(x={"x": train_data},
        y=train_labels,
        batch_size=32,
        num_epochs=1,
        shuffle=True)
classifier.train(input_fn=train_input_fn, steps=657)

score 0 · Answer 1 · answered Jun 20 '19 at 11:20

u can use tf.data.dataset. such as : tf.data.TextLineDataset or tf.data.TFRecordDataset.

then process the batch_size, epoch,shuffle in input_fn:

    def input_fn():

        dataset=tf.data.TextLineDataset(file_path)

        dataset=dataset.map(map_fn)
        dataset=dataset.shuffle(shuffle).repeat(epochs).batch(batch_size)
        return dataset

How to load only batch data to in memory during training with estimator?

1 Answers1