How can I flush GPU memory using CUDA (physical reset is unavailable)

Question

My CUDA program crashed during execution, before memory was flushed. As a result, device memory remained occupied.

I'm running on a GTX 580, for which nvidia-smi --gpu-reset is not supported.

Placing cudaDeviceReset() in the beginning of the program is only affecting the current context created by the process and doesn't flush the memory allocated before it.

I'm accessing a Fedora server with that GPU remotely, so physical reset is quite complicated.

So, the question is - Is there any way to flush the device memory in this situation?

"As a result, device memory remains occupied" - How do you know this to be true? — talonmies, Mar 04 '13 at 08:28
Although `nvidia-smi --gpu-reset` is not available, I can still get some information with `nvidia-smi -q`. In most fields it gives 'N/A', but some information is useful. Here is the relevant output: `Memory Usage Total : 1535 MB Used : 1227 MB Free : 307 MB` — timdim, Mar 04 '13 at 08:35
Plus, I fail to allocate memory for variables, which are small enough — timdim, Mar 04 '13 at 08:36
Is the process which was holding the context on the GPU still alive? Even catastrophic termination of a process should result in the driver destroying the context and releasing resources. — talonmies, Mar 04 '13 at 09:19
It doesn't look like it is alive. At least, I don't see it alive on CPU. I guess, the process on GPU cannot be alive as well, since I can launch another kernel (concurrent execution is not available on my GPU). But the memory is still occupied, I can be sure about it because of the reasons described above — timdim, Mar 04 '13 at 09:25
If you have root access, you can unload and reload the `nvidia` driver. — tera, Mar 04 '13 at 10:14
If you do `ps -ef |grep 'whoami'` and the results show any processes that appear to be related to your crashed session, kill those. (the single quote ' should be replaced with backtick ` ) — Robert Crovella, Mar 04 '13 at 16:18
ksooklall has a great answer to find what is hogging the memory, even if it doesn't show on nvidia-smi. — Davidmh, Oct 10 '17 at 18:50
`nvidia-smi -caa` worked great for me to release memory on all GPUs at once. — David Arenburg, Jun 25 '19 at 11:54

Kenan · Answer 1 · 2018-02-21T22:04:18.393

185

check what is using your GPU memory with

sudo fuser -v /dev/nvidia*

Your output will look something like this:

                     USER        PID  ACCESS COMMAND
/dev/nvidia0:        root       1256  F...m  Xorg
                     username   2057  F...m  compiz
                     username   2759  F...m  chrome
                     username   2777  F...m  chrome
                     username   20450 F...m  python
                     username   20699 F...m  python

Then kill the PID that you no longer need on htop or with

sudo kill -9 PID.

In the example above, Pycharm was eating a lot of memory so I killed 20450 and 20699.

edited Feb 21 '18 at 22:04

answered Oct 06 '17 at 02:07

Kenan

13,156
8
43
50

10

Thank you! For some reason, I had a process hogging all my VRAM, not showing on `nvidia-smi`. – Davidmh Oct 10 '17 at 18:48
1

I need to use this a lot when running deep learning in different jupyter notebooks. The only issue is knowing exactly which PID is which. Any tips on this? – Little Bobby Tables May 12 '18 at 21:56
is chrome --- the google chrome browser? if so what business does it have using a gpu? – kRazzy R Jun 05 '18 at 21:19
2

@josh I kill them one at a time making a mental note of the COMMAND. – Kenan Jun 07 '18 at 14:43
2

@kRazzyR - It uses it for speeding up computations, I assume, for rendering graphics, but maybe also other things. This did cause me a lot of issues when I install Nvidia drivers, CUDA and cudnn. I had to turn a lot of it off. See [here](https://www.lifewire.com/hardware-acceleration-in-chrome-4125122). – Little Bobby Tables Jun 07 '18 at 15:50
@ksooklall - I meant how do we know which jupyter notebook/process corresponds to which PID and therefore which one to kill... thanks. – Little Bobby Tables Jun 07 '18 at 15:52
I notice that I kill one then it start a new one. I tried to pkill -9 -t pts/1 to log out,still not working – Tina Liu May 31 '19 at 16:07
Try kill -9 or kill -15, not pkill – Kenan May 31 '19 at 17:37
1

In my case, `sudo` is not necessary. – one Aug 07 '20 at 00:58

score 61 · Answer 2 · answered Dec 14 '18 at 08:24

61

First type

nvidia-smi

then select the PID that you want to kill

sudo kill -9 PID

answered Dec 14 '18 at 08:24

Ashiq Imran

2,077
19
17

1

Brilliant, this one actually worked for me. PID should be replaced with the.. PID number of the process that uses the GPU (which you can figure by nvidia-smi) – Lior Magen Jul 26 '21 at 11:42
the command `nvidia-smi` returns `Failed to initialize NVML: Driver/library version mismatch` – desmond13 Nov 02 '21 at 10:40
nvidia-smi gives me two processes and when I go to kill them, it says no such process. processes are both called Xwayland – ejkitchen Oct 29 '22 at 00:21

talonmies · Accepted Answer · 2020-05-13T19:27:05.010

16

Although it should be unecessary to do this in anything other than exceptional circumstances, the recommended way to do this on linux hosts is to unload the nvidia driver by doing

$ rmmod nvidia

with suitable root privileges and then reloading it with

$ modprobe nvidia

If the machine is running X11, you will need to stop this manually beforehand, and restart it afterwards. The driver intialisation processes should eliminate any prior state on the device.

This answer has been assembled from comments and posted as a community wiki to get this question off the unanswered list for the CUDA tag

edited May 13 '20 at 19:27

answered May 18 '14 at 12:18

talonmies

70,661
34
192
269

8

cannot process the above command, error says, CUDA in use. So killed the PID using the solution provided by https://stackoverflow.com/a/46597252/3503565. Its works for me – DSBLR Apr 17 '19 at 00:53

score 14 · Answer 4 · answered Mar 31 '20 at 09:34

14

for the ones using python:

import torch, gc
gc.collect()
torch.cuda.empty_cache()

answered Mar 31 '20 at 09:34

Lukas

313
3
5

9

This cannot in any way to what the questioner was asking about – talonmies Jun 06 '20 at 11:31
nevertheless answered my problem (which is admittedly not the exact same as the OP asked, but matches the title while searching) – vmatyi Jan 10 '23 at 23:44
same here, any help is appreciated :) – Thiago Loureiro Feb 15 '23 at 19:15

score 11 · Answer 5 · edited Jul 19 '17 at 23:40

11

I also had the same problem, and I saw a good solution in quora, using

sudo kill -9 PID.

see https://www.quora.com/How-do-I-kill-all-the-computer-processes-shown-in-nvidia-smi

edited Jul 19 '17 at 23:40

Petter Friberg

21,252
9
60
109

answered Jul 19 '17 at 13:26

ailihong

127
1
2

Worked a treat when I accidentally opened and loaded two different *jupyter notebooks* with *VGG16*. **Warning**: it kills the notebooks. I guess you could pick one to free up some memory for the other but I dont know how you select the PID for a given notebook. – Little Bobby Tables Dec 30 '17 at 15:02

Antoine Viallon · Answer 6 · 2021-07-09T13:15:07.987

8

One can also use nvtop, which gives an interface very similar to htop, but showing your GPU(s) usage instead, with a nice graph. You can also kill processes directly from here.

Here is a link to its Github : https://github.com/Syllo/nvtop

NVTOP interface

edited Jul 09 '21 at 13:15

answered Apr 10 '20 at 09:57

Antoine Viallon

314
4
12

score 5 · Answer 7 · answered Oct 12 '16 at 21:54

on macOS (/ OS X), if someone else is having trouble with the OS apparently leaking memory:

https://github.com/phvu/cuda-smi is useful for quickly checking free memory
Quitting applications seems to free the memory they use. Quit everything you don't need, or quit applications one-by-one to see how much memory they used.
If that doesn't cut it (quitting about 10 applications freed about 500MB / 15% for me), the biggest consumer by far is WindowServer. You can Force quit it, which will also kill all applications you have running and log you out. But it's a bit faster than a restart and got me back to 90% free memory on the cuda device.

score 2 · Answer 8 · answered Dec 01 '21 at 05:01

2

For OS: UBUNTU 20.04 In the terminal type

nvtop

If the direct killing of consuming activity doesn't work then find and note the exact number of activity PID with most GPU usage.

sudo kill PID -number

answered Dec 01 '21 at 05:01

zahid hasan

29
1

AbuAli · Answer 9 · 2022-12-17T11:20:34.657

2

to kill all processess on GPU:

sudo fuser -v /dev/nvidia* -k

edited Dec 17 '22 at 11:20

answered Dec 17 '22 at 11:19

AbuAli

41
3

1

Instead of simply providing the answer directly, try writing a detailed comment that explains the solution, as long as the explanation is not too lengthy. @AbuAli . – DSDmark Dec 21 '22 at 15:08

score 0 · Answer 10 · answered Sep 28 '22 at 09:32

If you have the problem that after killing one process the next starts (Comment)- like for example when you have a bash script that calls multiple python scripts and you want to kill them but can't find its PID you can use ps -ef where you'll find the PID of your "problematic" process and also its PPID (parent PID). Use kill PPID or kill -9 PPID or sudo kill PPID to stop the processes.

score 0 · Answer 11 · edited Mar 02 '23 at 16:29

0

If all of this does not work, I found another answer here:

How to kill process on GPUs with PID in nvidia-smi using keyword?

nvidia-smi | grep 'python' | awk '{ print $X }' | xargs -n1 kill -9

Note that X (in the 'awk' expression) corresponds to the Xth column of your nvidia-smi command. If your nvidia-smi command looks like this, you should then replace X by 5.

edited Mar 02 '23 at 16:29

Fedor

17,146
13
40
131

answered Feb 24 '23 at 11:53

arnaudlenain

1
1

score 0 · Answer 12 · answered Jul 18 '23 at 21:09

Normally I just use nvidia-smi, but for some problems it's not enough (something still in cuda memory)

The nvidia-smi kill all is:

nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9

If you're still hitting unexpected memory errors or similar problems then try:

sudo fuser -v /dev/nvidia* | cut -d' ' -f2- | sudo xargs -n1 kill -9

score -2 · Answer 13 · answered Sep 18 '22 at 17:30

-2

I just started a new terminal and closed the old one and it worked out pretty well for me.

answered Sep 18 '22 at 17:30

kor

1
1

How can I flush GPU memory using CUDA (physical reset is unavailable)

13 Answers13

Linked