Running inference with a loaded tensorflow model in C++

Question

My tensorflow model for this example is simple and has been successfully loaded. My question is, to run inference, what do I pass the tf::Session? How do I create the feedDict mentioned here and what are my "outputOps"?

Here's the model in python:

model = tf.keras.Sequential([
tf.keras.layers.Convolution2D(filters, (5, 5), input_shape=(BoardDepth, BoardLength, BoardLength), data_format='channels_first', name='Input'),
tf.keras.layers.Activation('relu', name='Relu'),
tf.keras.layers.MaxPooling2D(pool_size=(2,2), name='Pool'),
tf.keras.layers.Flatten(name="Flatten"),
tf.keras.layers.Dense(BoardSize, activation='softmax', name='Output'),
])

optimizer = tf.train.AdamOptimizer(learning_rate=0.0018)

model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy']) 

Here's the method used to train the model which uses a generator to feed images in:

earlyExit = keras.callbacks.EarlyStopping(monitor='val_acc', min_delta=0, patience=0)

# Train the model
history = model.fit_generator(generator=gen.generator(),
                steps_per_epoch=gen.stepsPerEpoch(),
                validation_data=valGen.generator(),
                validation_steps=valGen.stepsPerEpoch(),
                epochs=numEpochs, 
                verbose=2, workers=1, callbacks=[earlyExit])

And here's the method used to save the model:

K.set_learning_phase(0)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver = tf.train.Saver(tf.trainable_variables())
    saver.save(sess, './models/test/latestModel')

In case you didn't follow the link above inference is supposed to be done with something like this:

// and run the inference to your liking
auto feedDict = ...
auto outputOps = ...
std::vector<tensorflow::Tensor> outputTensors;
status = session->Run(feedDict, outputOps, {}, &outputTensors);

I've tried following the example here but cannot seem to figure out what I need to pass as outputOps. Do I need to only pass the name of the output node, or do I need to pass every single operation in the model? The different configurations I've tried so far haven't worked.

My code so far looks something like this:

auto shape = tf::TensorShape({ BoardDepth, BoardSize, BoardSize});

// Placeholder until we're using actual input data
float inData[inputSize] = { 0.0 };

tf::Tensor input(tf::DT_FLOAT, shape);
std::copy_n(inData, inputSize, input.flat<float>().data());

std::vector<std::pair<std::string, tf::Tensor>> inputs = {
    { "Input/Conv2D", input },
};

std::vector<std::string> outOps = { "Output/bias:0"};

std::vector<tf::Tensor> outputs;

status = session->Run(inputs, outOps, {}, &outputs);

The input/output names are just set to names I could find that don't throw errors. I really don't know what I'm supposed to be setting them to in order for the model to function as it was trained.

Answer 1

It works exactly like the given example.py from the repository.

In python you write:

import tensorflow as tf
import numpy as np
sess = tf.Session()

from keras import backend as K
K.set_session(sess)

img = tf.placeholder(tf.float32, shape=(None, 2), name='input_plhdr')

model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', name='Intermediate'),
    tf.keras.layers.Dense(2, activation='softmax', name='Output'),
])

M = model(img)
print('input', img.name)
print('output', M.name)
sess.run(tf.global_variables_initializer())
print('result', sess.run(M, {img: np.array([[42, 43.]], dtype=np.float32)}))

saver = tf.train.Saver(tf.global_variables())
saver.save(sess, './exported/my_model')
tf.train.write_graph(sess.graph, '.', "./exported/graph.pb", as_text=False)

The

print('input', img.name)  # ('input', u'input_plhdr:0')
print('output', M.name)   # ('output', u'sequential/Output_1/Softmax:0')

tells you the name of the input and output. Adapting the inference_cc.cc you just need to fill in the correct names:

// same as in python file
auto data_ = data.flat<float>().data();
data_[0] = 42;
data_[1] = 43;

tensor_dict feed_dict = {
  { "input_plhdr", data },
};

std::vector<tensorflow::Tensor> outputs;
TF_CHECK_OK(sess->Run(feed_dict, {"sequential/Output_1/Softmax:0"}, {}, &outputs));

std::cout << "input           " << data.DebugString() << std::endl;
std::cout << "output          " << outputs[0].DebugString() << std::endl;

In Python I get

Using TensorFlow backend.
('input', u'input_plhdr:0')
('output', u'sequential/Output_1/Softmax:0')
('result', array([[1.0000000e+00, 1.8979743e-11]], dtype=float32))

And C++ returns

input           Tensor<type: float shape: [1,2] values: [42 43]>
output          Tensor<type: float shape: [1,2] values: [1 1.89797431e-11]>

Answer 2

Printing out the Output tensors name as Patwie's answer shows how to fix half the problem. The input tensors name for keras seems to be your first layer's name + _input, so in my case my input name was Input_input.