Tensorflow Different ways to Export and Run graph in C++

Question

For importing your trained network to the C++ you need to export your network to be able to do so. After searching a lot and finding almost no information about it, it was clarified that we should use freeze_graph() to be able to do it.

Thanks to the new 0.7 version of Tensorflow, they added documentation of it.

After looking into documentations, I found that there are few similar methods, can you tell what is the difference between freeze_graph() and: tf.train.export_meta_graph as it has similar parameters, but it seems it can also be used for importing models to C++ (I just guess the difference is that for using the file output by this method you can only use import_graph_def() or it's something else?)

Also one question about how to use write_graph(): In documentations the graph_def is given by sess.graph_def but in examples in freeze_graph() it is sess.graph.as_graph_def(). What is the difference between these two?

This question is related to this issue.

Thank you!

Answer 1

Here's my solution utilizing the V2 checkpoints introduced in TF 0.12.

There's no need to convert all variables to constants or freeze the graph.

Just for clarity, a V2 checkpoint looks like this in my directory models:

checkpoint  # some information on the name of the files in the checkpoint
my-model.data-00000-of-00001  # the saved weights
my-model.index  # probably definition of data layout in the previous file
my-model.meta  # protobuf of the graph (nodes and topology info)

Python part (saving)

with tf.Session() as sess:
    tf.train.Saver(tf.trainable_variables()).save(sess, 'models/my-model')

If you create the Saver with tf.trainable_variables(), you can save yourself some headache and storage space. But maybe some more complicated models need all data to be saved, then remove this argument to Saver, just make sure you're creating the Saver after your graph is created. It is also very wise to give all variables/layers unique names, otherwise you can run in different problems.

Python part (inference)

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('models/my-model.meta')
    saver.restore(sess, tf.train.latest_checkpoint('models/'))
    outputTensors = sess.run(outputOps, feed_dict=feedDict)

C++ part (inference)

Note that checkpointPath isn't a path to any of the existing files, just their common prefix. If you mistakenly put there path to the .index file, TF won't tell you that was wrong, but it will die during inference due to uninitialized variables.

#include <tensorflow/core/public/session.h>
#include <tensorflow/core/protobuf/meta_graph.pb.h>

using namespace std;
using namespace tensorflow;

...
// set up your input paths
const string pathToGraph = "models/my-model.meta"
const string checkpointPath = "models/my-model";
...

auto session = NewSession(SessionOptions());
if (session == nullptr) {
    throw runtime_error("Could not create Tensorflow session.");
}

Status status;

// Read in the protobuf graph we exported
MetaGraphDef graph_def;
status = ReadBinaryProto(Env::Default(), pathToGraph, &graph_def);
if (!status.ok()) {
    throw runtime_error("Error reading graph definition from " + pathToGraph + ": " + status.ToString());
}

// Add the graph to the session
status = session->Create(graph_def.graph_def());
if (!status.ok()) {
    throw runtime_error("Error creating graph: " + status.ToString());
}

// Read weights from the saved checkpoint
Tensor checkpointPathTensor(DT_STRING, TensorShape());
checkpointPathTensor.scalar<std::string>()() = checkpointPath;
status = session->Run(
        {{ graph_def.saver_def().filename_tensor_name(), checkpointPathTensor },},
        {},
        {graph_def.saver_def().restore_op_name()},
        nullptr);
if (!status.ok()) {
    throw runtime_error("Error loading checkpoint from " + checkpointPath + ": " + status.ToString());
}

// and run the inference to your liking
auto feedDict = ...
auto outputOps = ...
std::vector<tensorflow::Tensor> outputTensors;
status = session->Run(feedDict, outputOps, {}, &outputTensors);

Answer 2

For predicting (and every other operations) you can do something like this:

First of all in python you should name your variables or operation for the future use

self.init = tf.initialize_variables(tf.all_variables(), name="nInit")

After training, calculations of so.. when you have your variables assigned go through all of them and save as constants to your graph. (almost the same can be done with that freeze tool, but i usually do it by myself, check "name=nWeights" in py and cpp below)

def save(self, filename):
    for variable in tf.trainable_variables():
        tensor = tf.constant(variable.eval())
        tf.assign(variable, tensor, name="nWeights")

    tf.train.write_graph(self.sess.graph_def, 'graph/', 'my_graph.pb', as_text=False)

Now go c++ and load our graph and load variables from saved constants:

void load(std::string my_model) {
        auto load_graph_status =
                ReadBinaryProto(tensorflow::Env::Default(), my_model, &graph_def);

        auto session_status = session->Create(graph_def);

        std::vector<tensorflow::Tensor> out;
        std::vector<string> vNames;

        int node_count = graph_def.node_size();
        for (int i = 0; i < node_count; i++) {
            auto n = graph_def.node(i);

            if (n.name().find("nWeights") != std::string::npos) {
                vNames.push_back(n.name());
            }
        }

        session->Run({}, vNames, {}, &out);

Now you have all of your neural net weights or other variables loaded.

Similarly, you can perform other operations (remember about names?); make input and output tensors of proper size, fill input tensor with data and run session like so:

auto operationStatus = session->Run(input, {"put_your_operation_here"}, {}, &out);

Answer 3

For TensorFlow v2, it's suggested to use tensorflow::LoadSavedModel which can take model (SavedModel) saved via Python API mode.save(). So you don't need to use FreezeSavedModel and GrapeDef.

Suppose that your TensorFlow model files are saved in model/ directory:

#include <tensorflow/cc/saved_model/loader.h>
#include <tensorflow/cc/saved_model/tag_constants.h>
#include <tensorflow/cc/tools/freeze_saved_model.h>

using namespace std;
using namespace tensorflow;

//namespace tf = tensorflow;

int main() {

    const std::string export_dir = "model/";

    SavedModelBundle model_bundle;
    SessionOptions session_options = SessionOptions();
    RunOptions run_options = RunOptions();
    Status status = LoadSavedModel(session_options, run_options, export_dir, {kSavedModelTagServe},
                                &model_bundle);

    if (status.ok()) {
        std::cout << "Session successfully loaded: " << status;
    }
    else {
        std::cerr << "Failed: " << status;
    }
return 0;
}