Pytorch RuntimeError: CUDA out of memory with a huge amount of free memory

Question

While training the model, I encountered the following problem:

RuntimeError: CUDA out of memory. Tried to allocate 304.00 MiB (GPU 0; 8.00 GiB total capacity; 142.76 MiB already allocated; 6.32 GiB free; 158.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

As we can see, the error occurs when trying to allocate 304 MiB of memory, while 6.32 GiB is free! What is the problem? As I can see, the suggested option is to set max_split_size_mb to avoid fragmentation. Will it help and how to do it correctly?

This is my version of PyTorch:

torch==1.10.2+cu113

torchvision==0.11.3+cu113

torchaudio===0.10.2+cu113

Had a similar issue in colab. Solved by reducing per_device_train_batch_size to 8 instead of 64 inside TrainingArguments — ENDEESA, Sep 12 '22 at 10:19
Why is this even a problem? I tried to create a 512x512 image, but 8GB video RAM not sufficient - what kind of crappy Python implementation is that? — alexpanter, Apr 14 '23 at 19:46

dazzafact · Answer 1 · 2023-03-21T15:57:33.367

9

I wasted several hours until I discovered that reducing the batch size and resizing the width of my input image (image size) were necessary steps.

edited Mar 21 '23 at 15:57

answered Aug 14 '22 at 08:31

dazzafact

2,570
3
30
49

I've tried reducing the image and batch sizes to very small values. Now the memory required for allocation is only 30Mb. However, I'm still getting the same issue... any ideas? -- RuntimeError: CUDA out of memory. Tried to allocate 30.00 MiB (GPU 0; 6.00 GiB total capacity; 5.16 GiB already allocated; 0 bytes free; 5.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF – Bugz Nov 11 '22 at 08:17
YOu need a nvidia GPU >2gb – dazzafact Nov 12 '22 at 10:06
1

I have a 6Gb worth of GPU memory but it isn't being allocated. How do I get this allocated? – Bugz Nov 14 '22 at 05:55
What Script to you use? Github repository ? – dazzafact Nov 15 '22 at 06:10
i'm trying this setup. (https://rentry.org/SDInstallation) – Bugz Nov 16 '22 at 09:07
1

try decrease "ddim_steps", or other parameters. Read 7. Common Errors/Tips. If nothing works, use CPU. – dazzafact Nov 16 '22 at 20:38
Maybe you could also try tuning `max_split_size_mb` as mentioned here: https://stackoverflow.com/a/73809045/6655150 – Kostas Mouratidis Nov 28 '22 at 10:09

score 2 · Answer 2 · answered Apr 29 '22 at 23:59

I was trying this command:

python3 val.py --weights ./weights/yolov5l-xs-1.pt --img 1996 --data ./data/VisDrone.yaml

and I have a 24G Titan video Card.

Then I reduced the image size and worked for me. to:

python3 val.py --weights ./weights/yolov5l-xs-1.pt --img 1280  --data ./data/VisDrone.yaml

Results:

Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|████████████████████████████████| 18/18 [00:50<00:00,  2.79s/it]
                 all        548      38759      0.653      0.537      0.584      0.375
          pedestrian        548       8844       0.74      0.631      0.708      0.375
              people        548       5125      0.677      0.506      0.574      0.258
             bicycle        548       1287      0.541      0.377       0.41      0.213
                 car        548      14064      0.828      0.868      0.904      0.681
                 van        548       1975      0.636      0.566      0.601      0.453
               truck        548        750      0.595      0.516      0.538      0.388
            tricycle        548       1045      0.601      0.416      0.457      0.288
     awning-tricycle        548        532      0.387      0.242      0.245      0.173
                 bus        548        251      0.782      0.653      0.725      0.565
               motor        548       4886      0.744      0.598      0.674      0.355

score 1 · Answer 3 · edited Sep 03 '22 at 21:09

1

Your problem may be due to fragmentation of your GPU memory.You may want to empty your cached memory used by caching allocator.

import torch
torch.cuda.empty_cache()

edited Sep 03 '22 at 21:09

Elazar

20,415
4
46
67

answered Mar 16 '22 at 14:03

Erol Gelbul

25
3

5

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 16 '22 at 14:57
4

I called this method before training the model and got the same error. – Dmitrii Troshin Mar 16 '22 at 17:55
Can you try to decrease the batch size? And make sure you restart before running again. – Erol Gelbul Mar 17 '22 at 11:36
where do you put that command? Is it in the launch.py file? – hellwraiz Feb 12 '23 at 21:41
According to [this post](https://discuss.pytorch.org/t/keep-getting-cuda-oom-error-with-pytorch-failing-to-allocate-all-free-memory/133896/14), we shouldn't use `empty_cache()`. – Shengwei Jun 20 '23 at 15:52

score -4 · Answer 4 · answered Dec 06 '22 at 09:02

-4

It works for me with this:

pip install accelerate

answered Dec 06 '22 at 09:02

Albert Yu

19
1

5

Well, that import alone is very unlikely to be of any help whatsoever. – Matthias Urlichs Dec 09 '22 at 18:08

Pytorch RuntimeError: CUDA out of memory with a huge amount of free memory

4 Answers4

Linked