TensorFlow: How to predict from a SavedModel?

Question

I have exported a SavedModel and now I with to load it back in and make a prediction. It was trained with the following features and labels:

F1 : FLOAT32
F2 : FLOAT32
F3 : FLOAT32
L1 : FLOAT32

So say I want to feed in the values 20.9, 1.8, 0.9 get a single FLOAT32 prediction. How do I accomplish this? I have managed to successfully load the model, but I am not sure how to access it to make the prediction call.

with tf.Session(graph=tf.Graph()) as sess:
    tf.saved_model.loader.load(
        sess,
        [tf.saved_model.tag_constants.SERVING],
        "/job/export/Servo/1503723455"
    )

    # How can I predict from here?
    # I want to do something like prediction = model.predict([20.9, 1.8, 0.9])

This question is not a duplicate of the question posted here. This question focuses on a minimal example of performing inference on a SavedModel of any model class (not just limited to tf.estimator) and the syntax of specifying input and output node names.

Possible duplicate of [How to import an saved Tensorflow model train using tf.estimator and predict on input data](https://stackoverflow.com/questions/46098863/how-to-import-an-saved-tensorflow-model-train-using-tf-estimator-and-predict-on) — rhaertel80, Sep 13 '17 at 15:02
check detailed answer with tensorflow 1.7 here https://stackoverflow.com/a/52222383/5904928 — Aaditya Ura, Sep 07 '18 at 12:17

rhaertel80 · Answer 1 · 2017-09-14T07:12:04.707

15

Assuming you want predictions in Python, SavedModelPredictor is probably the easiest way to load a SavedModel and get predictions. Suppose you save your model like so:

# Build the graph
f1 = tf.placeholder(shape=[], dtype=tf.float32)
f2 = tf.placeholder(shape=[], dtype=tf.float32)
f3 = tf.placeholder(shape=[], dtype=tf.float32)
l1 = tf.placeholder(shape=[], dtype=tf.float32)
output = build_graph(f1, f2, f3, l1)

# Save the model
inputs = {'F1': f1, 'F2': f2, 'F3': f3, 'L1': l1}
outputs = {'output': output_tensor}
tf.contrib.simple_save(sess, export_dir, inputs, outputs)

(The inputs can be any shape and don't even have to be placeholders nor root nodes in the graph).

Then, in the Python program that will use the SavedModel, we can get predictions like so:

from tensorflow.contrib import predictor

predict_fn = predictor.from_saved_model(export_dir)
predictions = predict_fn(
    {"F1": 1.0, "F2": 2.0, "F3": 3.0, "L1": 4.0})
print(predictions)

This answer shows how to get predictions in Java, C++, and Python (despite the fact that the question is focused on Estimators, the answer actually applies independently of how the SavedModel is created).

edited Sep 14 '17 at 07:12

answered Sep 13 '17 at 22:59

rhaertel80

8,254
1
31
47

Evidently simple_save does is not compatible with graph building code when inputs to model are being read from input files using tf.data.Dataset and its iterator because simple_save requires tensors not numpy arrays. Furthermore the namespace is changed to tf.saved_model not tf.contrib. My code might be the problem. A known working code example using model trained using Dataset and saved with simple_save would be excellent to have @mrry. – Geoffrey Anderson Sep 07 '18 at 17:11
Evidently a further incompatibility of simple_save is with graph-making code that inputs sparse numpy arrays, and the first line of graph-making code is a tf.stack because it is a sparse matrix. So where do you put the tf.stack, outside the graph building code? A known working code example using a model calling tf.stack and saved with simple_save would be excellent to have @mrry – Geoffrey Anderson Sep 07 '18 at 17:14
@GeoffreyAnderson Sounds worthy of its own question; be sure to post snippets of the code you are using. – rhaertel80 Sep 09 '18 at 09:01
Just a note that this may no longer work in TensorFlow 2.0. At the very least, the imports may have changed. – Eric McLachlan Dec 15 '20 at 08:42

score 7 · Answer 2 · edited Dec 05 '17 at 02:59

For anyone who needs a working example of saving a trained canned model and serving it without tensorflow serving ,I have documented here https://github.com/tettusud/tensorflow-examples/tree/master/estimators

You can create a predictor from tf.tensorflow.contrib.predictor.from_saved_model( exported_model_path)

Prepare input

tf.train.Example( 
    features= tf.train.Features(
        feature={
            'x': tf.train.Feature(
                 float_list=tf.train.FloatList(value=[6.4, 3.2, 4.5, 1.5])
            )     
        }
    )    
)

Here x is the name of the input that was given in input_receiver_function at the time of exporting. for eg:

feature_spec = {'x': tf.FixedLenFeature([4],tf.float32)}

def serving_input_receiver_fn():
    serialized_tf_example = tf.placeholder(dtype=tf.string,
                                           shape=[None],
                                           name='input_tensors')
    receiver_tensors = {'inputs': serialized_tf_example}
    features = tf.parse_example(serialized_tf_example, feature_spec)
    return tf.estimator.export.ServingInputReceiver(features, receiver_tensors)

score 6 · Accepted Answer · answered Aug 27 '17 at 08:15

Once the graph is loaded, it is available in the current context and you can feed input data through it to obtain predictions. Each use-case is rather different, but the addition to your code will look something like this:

with tf.Session(graph=tf.Graph()) as sess:
    tf.saved_model.loader.load(
        sess,
        [tf.saved_model.tag_constants.SERVING],
        "/job/export/Servo/1503723455"
    )

    prediction = sess.run(
        'prefix/predictions/Identity:0',
        feed_dict={
            'Placeholder:0': [20.9],
            'Placeholder_1:0': [1.8],
            'Placeholder_2:0': [0.9]
        }
    )

    print(prediction)

Here, you need to know the names of what your prediction inputs will be. If you did not give them a nave in your serving_fn, then they default to Placeholder_n, where n is the nth feature.

The first string argument of sess.run is the name of the prediction target. This will vary based on your use case.

Why can't we pass multiple values to each placeholder, something like a batch or inputs? ```prediction = sess.run( 'prefix/predictions/Identity:0', feed_dict={ 'Placeholder:0': [20.9, 11.3], 'Placeholder_1:0': [1.8, 2.6], 'Placeholder_2:0': [0.9, 0.76] } )``` — Nitin, Nov 13 '19 at 20:03

score 0 · Answer 4 · edited Sep 07 '18 at 18:32

0

The constructor of tf.estimator.DNNClassifier has an argument called warm_start_from. You can give it the SavedModel folder name and it will recover your session.

edited Sep 07 '18 at 18:32

pushkin

9,575
15
51
95

answered Sep 07 '18 at 18:10

udifuchs

351
2
5

The `warm_start_from` folder contains checkpoint or `SavedModel`? They are not the same. – meelo Nov 29 '18 at 03:00

TensorFlow: How to predict from a SavedModel?

4 Answers4

Linked