4

I have a trained model that I want to use in a python application, but I can't see any examples of deploying to production environment without installing TensorFlow or creating gRPC service. Is it possible at all? What is the right approach in this situation?

hdt
  • 317
  • 2
  • 7
  • 1
    "Model" is a plain-text or binary description of a computational graph and a set of pre-computed parameters for this graph. How do you imagine to run the model without a library that implements operations of this graph? – Ivan Aksamentov - Drop Sep 16 '16 at 00:58
  • Maybe with some sort of object serialization? – hdt Sep 16 '16 at 06:41

2 Answers2

2

How do you want to serve it if not using TensorFlow itself or TensorFlow serving? Do you plan on reimplementing the TensorFlow operations to get the same semantics?

That said, with XLA there is now a way to compile a TensorFlow model into a binary which can be called from C++. See the documentation on tfcompile for an example.

Alexandre Passos
  • 5,186
  • 1
  • 14
  • 19
  • 1
    Thanks. I didn't know this feature. The links should be https://www.tensorflow.org/performance/xla/tfcompile – hdt Jun 27 '17 at 11:36
0

You can deploy a tensorflow model without tensorflow by using NVIDIA's TensorRT deep learning inference library, which is now compatible with tensorflow since version 3 of the library. It is tailored for inference so it is a very good choice if you fullfill its requirements.

However, it won't work for you if you plan to do inference on CPU or on a platform that is not supported by TensorRT (e.g. Windows).

P-Gn
  • 23,115
  • 9
  • 87
  • 104
  • loading the model needs tf [tf.saved_model.load](https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#samples) – AJ AJ Sep 22 '22 at 11:34