tensorflow : how to save/restore trained model

Question

this is an example how to save and restore a trained model. hope this would help for beginners.

generating 1 hidden layer neural network with relu activation function. (heard relu has been proved much better than sigmoid, especially for neural network with large number of hidden layer.)

training data is apparently XOR.

train and save "tf_train_save.py"

import tensorflow as tf
import numpy as np

x = np.matrix([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.matrix([[0], [1], [1], [0]])

n_batch = x.shape[0]
n_input = x.shape[1]
n_hidden = 5
n_classes = y.shape[1]

X = tf.placeholder(tf.float32, [None, n_input], name="X")
Y = tf.placeholder(tf.float32, [None, n_classes], name="Y")

w_h = tf.Variable(tf.random_normal([n_input, n_hidden], stddev=0.01), tf.float32, name="w_h")
w_o = tf.Variable(tf.random_normal([n_hidden, n_classes], stddev=0.01), tf.float32, name="w_o")

l_h = tf.nn.relu(tf.matmul(X, w_h))
hypo = tf.nn.relu(tf.matmul(l_h, w_o), name="output")

cost = tf.reduce_mean(tf.square(Y-hypo))
train = tf.train.GradientDescentOptimizer(0.1).minimize(cost)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(1000):
        for i in range(4):
            sess.run(train, feed_dict = {X:x[i,:], Y:y[i,:]})

    result = sess.run([hypo, tf.floor(hypo+0.5)], feed_dict={X:x})

    print(*result[0])
    print(*result[1])

    output_graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ["output"])
    tf.train.write_graph(output_graph_def, "./logs/mp_logs", "test.pb", False)

load "tf_load.py"

import tensorflow as tf
from tensorflow.python.platform import gfile
import numpy as np

x = np.matrix([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.matrix([[0], [1], [1], [0]])

with gfile.FastGFile("./logs/mp_logs/test.pb",'rb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())
    tf.import_graph_def(graph_def, name='')

with tf.Session() as sess:
    X = sess.graph.get_tensor_by_name("X:0")
    print(X)
    output = sess.graph.get_tensor_by_name("output:0")
    print(output)

    tf.global_variables_initializer().run()

    result = sess.run([output, tf.floor(output+0.5)], feed_dict={X:x})

    print(*result[0])
    print(*result[1])

would there be simpler way?

Your question title does not seem to match what you are asking. Assuming the title question, is your coding doing what you expect? I wonder about the initialization in the load script. — Eric Platon, Aug 31 '17 at 02:35
you dint save your weights variable nether you load them, so your code is incorrect . Look at this https://stackoverflow.com/questions/33759623/tensorflow-how-to-save-restore-a-model?rq=1 — Ishant Mrinal, Aug 31 '17 at 06:26
@EricPlaton It works. I was just curious if there are simpler way. like... saving tensor name too. — Jaewon.A.C, Aug 31 '17 at 06:56
@IshantMrinal It works unless I misread the result. the weights are converted into constants and no need to be handled cuz it's trained model. output_graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ["output"]) — Jaewon.A.C, Aug 31 '17 at 06:58
no, it only writes the graph, not the values. you are not getting errors cause of this line` tf.global_variables_initializer().run()` , it initialize values randomly. — Ishant Mrinal, Aug 31 '17 at 07:24
@Is it? cuz both 'last training result' and 'loaded graph's output' shows exactly same float value. I think there are no variables in loaded graph, since they're converted into constant before saved. — Jaewon.A.C, Aug 31 '17 at 07:43

score 2 · Accepted Answer · answered Aug 31 '17 at 08:16

2

You are using convert_variables_to_constants, so you are good indeed on the training side. Note for the passer-by, that API appeared in v1.0 (if I am not mistaken after tracking a bit the API).

On the load side, I think the minimum code is one command shorter. Given that you have turned all the variables into constants, there is no variable to initialize on restore. So the line:

tf.global_variables_initializer().run()

Does nothing. From the docs of v1.3:

If var_list is empty, however, the function still returns an Op that can be run. That Op just has no effect.

The load script has no global varibale, and since tf.global_variables_initializer() is equivalent to tf.variables_initializer(tf.global_variables()), the operation is a no-op.

answered Aug 31 '17 at 08:16

Eric Platon

9,819
6
41
48

1

I was expecting restore without handling name of tensor like 'input' and 'output'. couldn't find examples. I thought it's possible while reading VGGish source code. But I misread it. they just made a function that defines graph and used them both side in generating and restoring function. Guess I have to do same and handle the graph file and py file together – Jaewon.A.C Aug 31 '17 at 08:35
Is it possible to do this without converting variables to constants? Like if you want to continue training those variables after you load them? – Peter Nov 09 '18 at 11:45
1

@Peter Do you mean save/restore without passing by constants? Yes, that is the "textbook" case. Before recent changes moving toward Keras, the [programming guides from the TF homepage](https://github.com/tensorflow/docs/blob/r1.6/site/en/programmers_guide/saved_model.md) explained how to save/restore for retraining (that includes no conversion to constant). Note my link goes back to TF 1.6. If you go to the TF homepage now, it's quite different, so confusing in this thread based on an earlier API. This API still exists at this point. – Eric Platon Nov 11 '18 at 03:10

tensorflow : how to save/restore trained model

1 Answers1