How to kill process on GPUs with PID in nvidia-smi using keyword?

Question

How to kill running processes on GPUs for a specific program (e.g. python) in terminal? For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi

score 64 · Answer 1 · answered May 04 '19 at 16:21

64

The accepted answer doesn't work for me, probably because nvidia-smi has different formats across different versions/hardware.

I'm using a much cleaner command:

nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9

You can replace $3 in the awk expression to fit your nvidia-smi output. It is the n-th column in which the PIDs occur.

answered May 04 '19 at 16:21

Ainz Titor

1,497
1
15
22

2

Same here, accepted answer didn't work. This one did. Kudos !! – Pramesh Bajracharya Nov 25 '19 at 09:18
2

I needed to sudo kill `nvidia-smi | grep 'python' | awk '{ print $3 }' | sudo xargs -n1 kill -9` – Aaron Soellinger Dec 31 '19 at 10:52
hmm, this command gives me `kill: argument couldn't be read: '1-nan'` – MJimitater Jun 23 '20 at 23:12
1

`kill -9 ` works. use `sudo` wisely. – Prakash Vanapalli Feb 21 '21 at 04:44
4

had to change to awk '{ print $5}' but this worked – misinglink Apr 02 '21 at 08:11
5

for me, `nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9` worked. – Aaditya Ura Dec 06 '21 at 08:30

score 14 · Answer 2 · edited Jul 31 '21 at 15:17

14

Use nvidia-smi or top command to see processes running and to kill command:

sudo kill -9 PID

edited Jul 31 '21 at 15:17

MD Mushfirat Mohaimin

1,966
3
10
22

answered Jul 30 '21 at 12:31

Snehal Rajput

335
3
6

This answer is addressed as a comment in response to @oya163 under the accepted answer. – salehinejad Aug 09 '21 at 01:45

score 11 · Accepted Answer · edited Aug 30 '23 at 08:30

11

You can grep "python" in the nvidia-smi and then pass the PID to the kill -9 command, e.g.

sudo kill -9 $( nvidia-smi | grep 'python' | sed -n 's/|\s*[0-9]*\s*\([0-9]*\)\s*.*/\1/p' |  sed '/^$/d')

edited Aug 30 '23 at 08:30

doneforaiur

1,308
7
14
21

answered May 05 '18 at 20:01

salehinejad

7,258
3
18
26

Where do I mention PID?? – oya163 Oct 15 '18 at 20:30
@oya163 it should grab it using grep from nvidia-smi. No need to manually add it. – salehinejad Oct 16 '18 at 21:03
No I meant a user-specific PID? Because I don't want to and I can't kill other's PID. – oya163 Oct 17 '18 at 20:57
9

@oya163 in this case you may try: sudo kill -9 PID – salehinejad Oct 18 '18 at 21:53
had to change to awk '{ print $5}' but this worked. – misinglink Apr 02 '21 at 08:09

score 9 · Answer 4 · answered Sep 24 '21 at 06:21

I guess the question is already answered when nvidia-smi shows processes occupying GPU mem. For me, even though nvidia-smi wasnt showing any processes, GPU memory was being used and I wanted to kill them.

The way to go in this case was to use the fuser command to find out the processes using the particular GPU device. In my case I wanted to kill all the processes using the GPU device 3. This can be done using the command :

sudo fuser -k /dev/nvidia3

You can use -ki to kill the processes interactively.

score 6 · Answer 5 · answered May 16 '22 at 22:28

As one of other answers suggest you can use: (replace 5 with the column number where process id exists)

nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9

If you might have to use this a lot you can create an alias for the command: to do that do this you should edit ~/.bash_aliases file:

nano ~/.bash_aliases

and add the following line to it and save the file:

alias killgpuprocess="nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9"

then (just needed this time):

source ~/.bashrc

Then if you run

killgpuprocess

it will kill the existing processes on GPU(s).

I had to use 5 instead of 3 for the top-voted answer to work. — Atif Ali, Sep 02 '22 at 06:53

score 0 · Answer 6 · answered Aug 30 '23 at 05:16

0

Keeping here as a reference. In case one wants to kill all running processes on the gpu, following command would work:

nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -n1 kill -9

answered Aug 30 '23 at 05:16

Bitswazsky

4,242
3
29
58

How to kill process on GPUs with PID in nvidia-smi using keyword?

6 Answers6

Linked