0

During runtime my model starts running out of memory (about 3k iterations in) with batches (all of size 64 exactly).

It did not run OOM during the initial iterations with the equivalent batches.

I run tensorboard summeries, Adam optimizer and use the while-loop for RNNs.

What are known causes of memory increase while running a model, and how can I keep track of memory usage? Normally tensorboard just occupies full memory.

Alexander R Johansen
  • 2,737
  • 3
  • 18
  • 23

1 Answers1

1

You can track the memory usage with Timeline.

I don't know why memory could be accumulating along with iterations. A possible workaround is to save the models to disk before out-of-memory and then restore to continue training.

Community
  • 1
  • 1
Yao Zhang
  • 5,591
  • 1
  • 16
  • 22