1

I am trying to train a custom model on weights in the darknet and the algorithm involved is yolov4. After the model is successfully loaded, I am getting a Cuda error: out of memory as shown below.

896 x 896
Create 6 permanent cpu-threads
Try to set subdivisions=64 in your cfg-file.
CUDA status Error: file: D:\darknet\src\dark_cuda.c : cuda_make_array() : line: 492 : 
build time: Jan 21 2022 - 16:57:15

CUDA Error: out of memory

As suggested in the error I even changed my subdivision=64 in the configuration file, but still I am getting the same error. I have tried various combinations of batch and subdivisions,but I am unable to solve this issue. I am using cuda version:10.1 and nvidia-gtx1050.

A snapshot of my configuration file:

enter image description here

Nimantha
  • 6,405
  • 6
  • 28
  • 69
YAMA
  • 11
  • 2
  • Are you using darknet-53 as a backbone? The operations with your configuration can use quite a lot of GPU memory. I would either try to bring batch and subdivisons down to 32 or 16 and see if I still run out of memory, or try scaling down the image size to 416x416 for example. – jiipeezz Jan 21 '22 at 14:00
  • It works after scaling down the image size to 416x416. Thank you! – YAMA Jan 28 '22 at 04:15
  • You can also see an explanation [here](https://stackoverflow.com/a/75930463/21565398). – Priyanka Choudhary Apr 04 '23 at 17:53

2 Answers2

0

if error Out of memory occurs then in .cfg-file you should increase subdivisions=16, 32 or 64

Abbas Shafi
  • 1
  • 1
0

See your resource utilization when you start training and see if the RAM causes it to exceed. If it is so then try this solution:

CFG-Parameters in the [net] section:

[net] section

batch - number of samples (images, letters, ...) which will be precossed in one batch

subdivisions- number of mini_batches in one batch, size mini_batch = batch/subdivisions,so GPU processes mini_batch samples at once, and the weights will be updated for batch samples (1 iteration processes batch images)

With reference from this,

I tried for min_batch=2 in google colaband that worked only So I consider the subdivisions half of the batch like:

batch=64 
subdivisions=32

"OR"

batch=32
subdivisions=16

Or any other...

Priyanka Choudhary
  • 1
  • 1