YOLOv4 detection in OpenCV.js causing Uncaught exception

Question

I'm trying to get YOLOv4 detection to work using OpenCV.js.

It looks like I've successfully gotten the net read using

const net = cv.readNetFromDarknet(files.config, files.weights);

And I'm using getBlobFromImage from the tutorial to get the network input:

getBlobFromImage = function(inputSize, mean, std, swapRB, image) {
    let mat;
    if (typeof(image) === 'string') {
        mat = cv.imread(image);
    } else {
        mat = image;
    }

    let matC3 = new cv.Mat(mat.matSize[0], mat.matSize[1], cv.CV_8UC3);
    cv.cvtColor(mat, matC3, cv.COLOR_RGBA2BGR);
    let input = cv.blobFromImage(matC3, std, new cv.Size(inputSize[0], inputSize[1]),
                                 new cv.Scalar(mean[0], mean[1], mean[2]), swapRB);

    matC3.delete();
    return input;
}

However, when I try to run forward on the network, I get an error that says Uncaught followed by a seemingly random number.

I've tried

net.setInput(blob)
net.forward()

And I get the error.

I found a source online that said you need to use layer names and an output vector using forward2, so I've tried that with hardcoded layer names (as OpenCV.js doesn't expose a method to get layer names):

net.setInput(blob)
const outs = new cv.MatVector();
net.forward2(outs, ["yolo_139", "yolo_150", "yolo_161"]);

And I still get the uncaught error. I've also resized my image so width and height are both multiples of 32, which I read somewhere was necessary.

Any ideas? I've been searching for answers but haven't found anything yet.

Edit: More detail on error

I just tried running it a few times, and here is the output:

Uncaught 279968808
Uncaught 279899688
[then, after refresh]
Uncaught 279968808
Uncaught 279899688

Here's the stack trace:

___cxa_throw opencv.js:30
<anonymous> opencv.js line 30 > WebAssembly.instantiate:899300
<anonymous> opencv.js line 30 > WebAssembly.instantiate:898391
<anonymous> opencv.js line 30 > WebAssembly.instantiate:315351
<anonymous> opencv.js line 30 > WebAssembly.instantiate:315293
<anonymous> opencv.js line 30 > WebAssembly.instantiate:545156
<anonymous> opencv.js line 30 > WebAssembly.instantiate:525900
<anonymous> opencv.js line 30 > WebAssembly.instantiate:525121
<anonymous> opencv.js line 30 > WebAssembly.instantiate:65086
<anonymous> opencv.js line 30 > WebAssembly.instantiate:65049
<anonymous> opencv.js line 30 > WebAssembly.instantiate:5596483
dynCall_iiiii opencv.js:30
dynCall_iiiii_9 opencv.js line 30 > Function:4
constructor_body opencv.js:30
constructor opencv.js:30
Mat opencv.js line 30 > Function:4
matFromImageData opencv.js:30
imread opencv.js:30
getBlobFromImage DetectApp.js:24
run DetectApp.js:63
<anonymous> debugger eval code:1

So that's an instance of it failing to run getBlobFromImage, but that only fails occasionally, it usually succeeds. Here's the more common stack trace, with a failure in dnn_Net$forward2:

___cxa_throw opencv.js:30
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:899300
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:898391
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:315351
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:315293
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:545156
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:525900
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:526317
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:697106
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:327453
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:529953
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:2720179
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:2717263
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:2717095
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:3384719
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:3323788
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:3318607
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:3365552
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:2646791
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:141167
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:204282
    <anonymous> opencv.js line 30 > WebAssembly.instantiate:5596551
    dynCall_viiii opencv.js:30
    dynCall_viiii_1868 opencv.js line 30 > Function:4
    dnn_Net$forward2 opencv.js line 30 > Function:10
    run DetectApp.js:66
    <anonymous> debugger eval code:1

"seemingly random number", in programming, is often just a _code_ for something, i.e. *not random at all*. — Christoph Rackwitz, Jun 25 '22 at 09:02
Fair enough. I just tried it 4 times, and the errors are as follows: "Uncaught 279968808", "Uncaught 279899688", "Uncaught 279865576", "Uncaught 279797736". So clearly some pattern as it has a fixed beginning and decreasing somewhere between 30 and 70 thousand each time. But no error code that I could find online. — Nathan, Jun 25 '22 at 17:06
I've added the stack traces to the question. Any advice here would be helpful, I'm pretty stuck. — Nathan, Jun 25 '22 at 17:18
I don't see why images have to be sized a multiple of 32, but they _do_ need to be sized as the network wants it. google says it's 224x224x3 for yolov4 -- if you try an equivalent program (every instruction is equivalent) in python or c++, does that work? if yes, it might be an issue with how OpenCV is transpiled to work on javascript -- compare what you do to whatever opencv.js dnn examples can be found in the docs or in the repository among the samples — Christoph Rackwitz, Jul 04 '22 at 08:31
@ChristophRackwitz do you have any links on that? I followed a darknet training tutorial and it didn't say anything about sizes for training data, and the python OpenCV yolov4 detection I've run seemed to work on variable sized images. — Nathan, Jul 04 '22 at 15:44
"seemed to work" because something was resizing the inputs. I don't see that here. — Christoph Rackwitz, Jul 04 '22 at 15:57
@ChristophRackwitz I'll look into it, thank you. If you post this as an answer (and link to any resources you have on it) I can award you the bounty. — Nathan, Jul 04 '22 at 17:53
I'm just guessing/probing here and it wouldn't feel right to take any credit for something that I don't know is actually the solution. thanks anyway. — Christoph Rackwitz, Jul 04 '22 at 19:48
@ChristophRackwitz well, your advice seems to have worked! I resized to 416x416 (which is what the python script that works is doing), and forward2 seems to run and produce output. It's pretty darn slow though, so my hopes of realtime webcam detection seem dashed unfortunately. — Nathan, Jul 04 '22 at 21:18
(@ChristophRackwitz since the answer works, post an answer if you feel inclined and I'll award you the bounty, you earned it :) ) — Nathan, Jul 04 '22 at 21:53
yeah full size yolo is made for GPUs and I have no idea whether the OpenCV dnn stuff can use GPUs after being transpiled into javascript. should be possible in principle (there's "gpu.js" for gpgpu in the browser). perhaps try a "tiny yolo" flavor. — Christoph Rackwitz, Jul 05 '22 at 07:50

score 1 · Accepted Answer · answered Jul 05 '22 at 07:47

The error messages are quite opaque. If you're running on the latest build and still this crashes without clear error messages, perhaps submit an issue on OpenCV's github but check first if that hasn't been reported already.

I did not see any resizing in your inference code. If the inference code you have doesn't already resize inputs, you should try adding that.

Neural networks usually expect inputs of a specific size*. For YOLOv4 that might be 416x416 or 224x224 (tiny?), which are determined when the network is designed.

During training, either all the inputs are already the right size, or the training script resizes the inputs.

* fully convolutional networks take arbitrary sizes

score 0 · Answer 2 · answered Jul 04 '22 at 02:00

The Uncaught numbers aren't random numbers, they are pointers. Usually problem like this happen when the classifier hasn't been correctly configured.

Make sure config and weights parameters are valid, also try to explicit the backend choice with

net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV)

YOLOv4 detection in OpenCV.js causing Uncaught exception

2 Answers2

Linked