1

I'm trying to load a simple four-layer convolutional neural network from an ONNX file in C++ with OpenCV. The ONNX file was created from a TensorFlow model using the tf2onnx library in Python. I saved the model with the following piece of code.

(onnx_model_proto, storage) = tf2onnx.convert.from_keras(model, opset=8)
with open(os.path.join("models", 'upscaleModelData.onnx'), "wb") as f:
    f.write(onnx_model_proto.SerializeToString())

When reading via cv::dnn::Net net = cv::dnn::readNetFromONNX("model.onnx"); in C++, I get the following error.

[ INFO:0] global ####\master_winpack-build-win64-vc15\opencv\modules\dnn\src\onnx\onnx_importer.cpp (429) cv::dnn::dnn4_v20210608::ONNXImporter::populateNet DNN/ONNX: loading ONNX v4 model produced by 'tf2onnx':1.10.0. Number of nodes = 13, inputs = 1, outputs = 1
OpenCV(4.5.3) Error: Unspecified error (Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace") in cv::dnn::dnn4_v20210608::LayerData::getLayerInstance, file ####\opencv\modules\dnn\src\dnn.cpp, line 621
[ERROR:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2127) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0)
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode     Input[0] = 'model/conv2d_3/Relu:0'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2135) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode     Output[0] = 'model/tf.nn.depth_to_space/DepthToSpace:0'
OpenCV(4.5.3) Error: Unspecified error (> Node [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:621: error: (-2:Unspecified error) Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace" in function 'cv::dnn::dnn4_v20210608::LayerData::getLayerInstance'
> ) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 2146
OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2146: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> Node [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:621: error: (-2:Unspecified error) Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace" in function 'cv::dnn::dnn4_v20210608::LayerData::getLayerInstance'

I'm using OpenCV on Windows and Visual C++.

This might be a similar issue to this: EMGU - EDSR : Can't create layer DepthToSpace.

I tried reading the same network from TensorFlow PB and H5 file, but I get a similar result.

For reading a PB file, created with cv::dnn::readNetFromTensorflow("model.pb") I get:

OpenCV(4.5.3) Error: Unspecified error (FAILED: fs.is_open(). Can't open "model.pb") in cv::dnn::ReadProtoFromBinaryFile, file ####\opencv\modules\dnn\src\caffe\caffe_io.cpp, line 1133
OpenCV(4.5.3) ####\opencv\modules\dnn\src\caffe\caffe_io.cpp:1133: error: (-2:Unspecified error) FAILED: fs.is_open(). Can't open "model.pb" in function 'cv::dnn::ReadProtoFromBinaryFile'

For reading a H5 file created with cv::dnn::readNetFromTensorflow("upscaleModelData.h5"); I get:

OpenCV(4.5.3) Error: Unspecified error (Cannot determine an origin framework of files: model.h5) in cv::dnn::dnn4_v20210608::readNet, file ####\opencv\modules\dnn\src\dnn.cpp, line 5461
OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:5461: error: (-2:Unspecified error) Cannot determine an origin framework of files: model.h5 in function 'cv::dnn::dnn4_v20210608::readNet'

Does this mean I should make modifications to the model's layers so that it can be read by OpenCV? Is this a compatibility issue? Any feedback, workarounds or alternative approaches (e.g. TensorFlow C++ API solutions) are welcome.

Edit 1: Implementing a layer-type workaround

I reimplemented the depth_to_space layer manually in Python, using the following piece of code, based on these links: onnx-tensorflow, depth_to_space, keras-subpixel-conv.

x_shape = tf.shape(x)
n, h, w, c = x_shape[0], x_shape[1], x_shape[2], x_shape[3]

y = tf.reshape(x, (n, h, w, bs, bs, c // (bs ** 2)))
y = tf.transpose(y, (0, 1, 3, 2, 4, 5))
outputs = tf.reshape(y, (n, h * bs, w * bs, c // (bs ** 2)))

Now, when I run the same C++ code, I get the following error.

[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (429) cv::dnn::dnn4_v20210608::ONNXImporter::populateNet DNN/ONNX: loading ONNX v4 model produced by 'tf2onnx':1.10.0. Number of nodes = 38, inputs = 14, outputs = 1
OpenCV(4.5.3) Error: Assertion failed (indexMat.total() == 1) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 1842
[ERROR:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2127) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Gather]:(model/tf.compat.v1.shape/Shape:0)
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode     Input[0] = 'Shape__72:0'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode     Input[1] = 'Const__76'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2135) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode     Output[0] = 'model/tf.compat.v1.shape/Shape:0'
OpenCV(4.5.3) Error: Unspecified error (> Node [Gather]:(model/tf.compat.v1.shape/Shape:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:1842: error: (-215:Assertion failed) indexMat.total() == 1 in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> ) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 2146
OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2146: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> Node [Gather]:(model/tf.compat.v1.shape/Shape:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:1842: error: (-215:Assertion failed) indexMat.total() == 1 in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
Igor Ševo
  • 5,459
  • 3
  • 35
  • 80
  • 1
    "Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace" in function 'cv::dnn::dnn4_v20210608::LayerData::getLayerInstance'" What's the type of that layer? Can you present the network architecture? – Micka Sep 17 '21 at 08:56
  • 1
    probably this layer type isnt implemented in OpenCV: https://github.com/onnx/onnx/blob/master/docs/Operators.md#DepthToSpace – Micka Sep 17 '21 at 08:57
  • 1
    are you using opencv with the contrib module? It looks like there is a DepthToSpace layer type in contrib: https://github.com/opencv/opencv_contrib/blob/master/modules/dnn_superres/src/dnn_superres.cpp – Micka Sep 17 '21 at 08:58
  • This is the layer type in question: https://www.tensorflow.org/api_docs/python/tf/nn/depth_to_space. I can relatively trivially fix this in the network, if that is the issue. – Igor Ševo Sep 17 '21 at 08:58
  • 1
    @Micka: I only downloaded basic prebuilt binaries. This could be the solution. – Igor Ševo Sep 17 '21 at 09:00
  • 1
    you could even try to compile that layer source code with your program, without adding the whole contrib code to your opencv. That DepthToSpace seems to be linked to the superres namespace in contrib. Not sure whether this will bring additional problems. You should try to check whethe the DepthToScale in superres is the same as the general one. If you could redefine your network architecture to not use depthToScale layer type at all, that might be the most simple fix. – Micka Sep 17 '21 at 09:05
  • I get the following error: terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.1.0) /home/Downloads/opencv-4.1.0/modules/dnn/src/dnn.cpp:524: error: (-2:Unspecified error) Can't create layer "StatefulPartitionedCall/inception_resnet_v1/AvgPool/Mean_Squeeze__1416:0" of type "Squeeze" in function 'getLayerInstance' Aborted (core dumped) – fisakhan Oct 19 '21 at 15:05
  • Can you please answer https://stackoverflow.com/questions/69633595/load-onnx-model-in-opencv-dnn?noredirect=1#comment123084665_69633595 – fisakhan Oct 21 '21 at 12:19

0 Answers0