Using Tensorflow 2.X model on OpenCV

Question

I have to use a Tensorflow 2.X model with the OpenCV framework (v.4.X with C++).

To do this, I need a single .pb file or a .pb and a .pbtxt file, instead of a Tensorflow Saved Model like the one I have.

So my question is: Is there a way to convert a Saved Model in a format that OpenCV could read? Like, maybe, a caffe model?

I tried with MMdnn but it gives me a strange error:

Traceback (most recent call last):
  File "/usr/local/bin/mmconvert", line 8, in <module>
    sys.exit(_main())
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/convert.py", line 102, in _main
    ret = convertToIR._convert(ir_args)
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/convertToIR.py", line 62, in _convert
    from mmdnn.conversion.tensorflow.tensorflow_parser import TensorflowParser
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/tensorflow/tensorflow_parser.py", line 15, in <module>
    from tensorflow.tools.graph_transforms import TransformGraph
ImportError: No module named 'tensorflow.tools.graph_transforms'

And I suppose it is because it was developed and tested with Tensorflow 1.X.

Edit: I also have the relative Keras model (now that it is integrated with Tensorflow 2), but it is incompatible with OpenCV DNN framework too. Trying converting it with MMdnn I get this error:

Traceback (most recent call last):
  File "/usr/local/bin/mmconvert", line 8, in <module>
    sys.exit(_main())
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/convert.py", line 102, in _main
    ret = convertToIR._convert(ir_args)
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/convertToIR.py", line 46, in _convert
    parser = Keras2Parser(model)
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/keras/keras2_parser.py", line 126, in __init__
    model = self._load_model(model[0], model[1])
  File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/keras/keras2_parser.py", line 78, in _load_model
    'DepthwiseConv2D': layers.DepthwiseConv2D})
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/saving.py", line 664, in model_from_json
    return deserialize(config, custom_objects=custom_objects)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/__init__.py", line 168, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/generic_utils.py", line 147, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 1056, in from_config
    process_layer(layer_data)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 1042, in process_layer
    custom_objects=custom_objects)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/__init__.py", line 168, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/generic_utils.py", line 149, in deserialize_keras_object
    return cls.from_config(config['config'])
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/base_layer.py", line 1179, in from_config
    return cls(**config)
  File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/convolutional.py", line 484, in __init__
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/convolutional.py", line 117, in __init__
    self.kernel_initializer = initializers.get(kernel_initializer)
  File "/usr/local/lib/python3.5/dist-packages/keras/initializers.py", line 515, in get
    return deserialize(identifier)
  File "/usr/local/lib/python3.5/dist-packages/keras/initializers.py", line 510, in deserialize
    printable_module_name='initializer')
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    ': ' + class_name)
ValueError: Unknown initializer: GlorotUniform

Edit 04/2021: Now the ONNX converter mentioned in the comments works properly with OpenCV 4.5.1 (Version 4.5.0 has a bug with some ONNX networks).

i guess ur getting this issue because u have tf 2.x on ur machine to erase this issue please install tf1.14 — Rajnish Kumar, Mar 06 '20 at 11:02
I tried it, but with tf 2.x i have a folder instead of a .pb and a .pbtxt; therefore it is not compatible with MMdnn. I also have a .h5 model (a keras model, that now it is integrated with tf 2.x), but it could not recognize the initializer GlorotUniform — Doch88, Mar 06 '20 at 11:44
What type of model are you trying to get working? NLP/Object Detection/Something else. And why does it need to be in openCV and not just use the tf 2.0 inference? An alternative might be converting to an ONNX model and using that in openCV. But if I know more about what you're trying to get at, I might be able to help more! — Victor Sonck, Mar 10 '20 at 10:53
It is an encoder-decoder network for a task similar to image segmentation; it is on specifications of the project to avoid using other libraries than OpenCV, but if there is **absolutely** no alternative I guess I must consider it. I didn't know about ONNX, can I easily convert a model from tf2 to ONNX? — Doch88, Mar 10 '20 at 11:21
With your H5 models it looks as if in theory it would work [just fine](https://github.com/onnx/keras-onnx). ONNX stands for the "Open Neural Network Exchange" and they're trying to alleviate the exact problem you have now. Let me know if it works! — Victor Sonck, Mar 10 '20 at 11:44
Ok, I tried with Keras2ONNX to convert the .h5 file to a .onnx file, but I got this error: **RuntimeError: tf2onnx loading failed with the tensorflow package!** Searching on Google I found this issue: https://github.com/onnx/tensorflow-onnx/issues/691 I guess that even if the model is a .h5 there is something using a tf2 feature that is not completely supported yet — Doch88, Mar 10 '20 at 13:07
Damn it. And how about converting from the h5 file to caffe? https://github.com/uhfband/keras2caffe In my experience the caffe support of openCV is quite good. Hopefully your model is simple enough that it can be as easy as this: https://github.com/uhfband/keras2caffe/blob/master/examples/face_classification/convert.py — Victor Sonck, Mar 10 '20 at 14:32
There is two ways of using keras. Install keras separately or use keras packaged with tensorflow. You should use tf.keras.models.load_model rather than keras.models.load_model. [see this link](https://stackoverflow.com/questions/53183865/unknown-initializer-glorotuniform-when-loading-keras-model) — nkvns, Mar 10 '20 at 14:33
@nkvns these two ways are based on the version of TensorFlow that you're using (1.X or 2.X), MMdnn is developed with tf 1.X and therefore it is using Keras "stand-alone", so I don't have much choice — Doch88, Mar 10 '20 at 15:56
@VictorSonck it seems that keras2caffe doesn't recognize the layer type "Conv2DTranspose" and so it cannot convert it. However, I'll check if I can implement this layer translation into keras2caffe — Doch88, Mar 10 '20 at 16:38
Now that you have the .h5 file, can you try [this approach](https://stackoverflow.com/a/45466355/5179465) instead of MMdnn? You can load the pretrained model with `model = load_model('./model/keras_model.h5')` before you freeze the graph. — ilke444, Mar 10 '20 at 19:37
@ilke444 after a lot of tries it seems to work. I had some problems with OpenCV but with some changes to the model (like using optimize_for_inference) it loads the model successfully. I only have to try if the inference works well and then I can say that the problem is solved, thank you very much — Doch88, Mar 11 '20 at 09:58
explained so well here look https://www.tensorflow.org/js/guide/conversion — Black Snow, Mar 11 '20 at 13:14
@BlackSnow the guide that you have linked explains how to convert a model to be compatible with tensorflow.js, it doesn't help me — Doch88, Mar 11 '20 at 18:22
@ilke444 predictions look fine. If you want you can answer this question and get your bounty, thank you! — Doch88, Mar 11 '20 at 20:25

score 3 · Accepted Answer · answered Mar 11 '20 at 23:08

If you have the .h5 file, you can try this approach instead of MMdnn, using TensorFlow. The function converts the current session into a static computation graph to capture current states. Then you can write the graph in .pb format using tf.train.write_graph.

You can load the pretrained model with model = load_model('./model/keras_model.h5') before you freeze the graph. There is also a blog post for further explanation.

Using Tensorflow 2.X model on OpenCV

1 Answers1