Can someone tell my why when I train my model using tensorflow-gpu in the jupyter notebook that my dedicated GPU memory is 85% in use even after the training model has completed so if I try to run the same model or a modified model I get the error Failed to get convolution algorithm. This is probably because cuDNN failed to initialize. If I want to run another model I need to quit the Anaconda Prompt and relaunch jupyter notebook for the memory to clear. Is this happening to anyone else? Does anyone know how to clear the GPU memory?
Asked
Active
Viewed 373 times
0
-
This is a long well known problem in Tensorflow and there are answers. Please check these links out: https://github.com/tensorflow/tensorflow/issues/17048 and https://stackoverflow.com/questions/59363874/cuda-error-out-of-memory-python-interpreter-utilizes-all-gpu-memory – velociraptor11 Jan 08 '20 at 01:35
-
Does this answer your question? [CUDA Error: out of memory - Python process utilizes all GPU memory](https://stackoverflow.com/questions/59363874/cuda-error-out-of-memory-python-process-utilizes-all-gpu-memory) – talonmies Jan 07 '21 at 12:09
1 Answers
0
You can do this:
pip install numba
from numba import cuda
cuda.select_device(<index of GPU>) # eg.cuda.select_device(0)
cuda.close()
Your Jupyter notebook will be unaffected but the command will kill the tf session that's in the background and so clear all the GPU memory. Note that because it clears the tf session you can't run this intermittently during a job to clear up memory as you go. Also, use pip not conda, not sure why but the former works, the latter seems to have a conflict ... but that could be just a version problem.
MikeB2019x
- 823
- 8
- 23