2

In tensorflow 1.14, it's obvious that tf.compat.v1.train.init_from_checkpoint can load ckpt to continue training (or to warm start). However, I couldn't find any corresponding approaches in SavedModel, and tf.estimator.WarmStartSetting only supports ckpt as well. It's weird for me, because this answer mentioned that there should be a checkpoint stored in SavedModel. Does anyone know:

  1. How to load checkpoint in SavedModel? or
  2. How to warm start training on SavedModel?
davislf2
  • 79
  • 10
  • If you are having model code with you, it is better to do saving and restoring using checkpoints.``` https://www.tensorflow.org/guide/checkpoint``` . It is even compatible with tf1.0 to load old checkpoints saved using ```saver.save()``` – Sarath R Nair Oct 09 '19 at 06:58
  • Thank you @SarathRNair. I know that checkpoints work. However, I'd like to deploy my model on tf-serving, which requires SavedModel format. That's why I'd like to know how to warm start from SavedModel, because I don't want to save and load the model twice (checkpoint and SavedModel). – davislf2 Oct 09 '19 at 07:16
  • Have you got a solution to this? – Sarath R Nair Oct 15 '19 at 06:09
  • 1
    Hi @SarathRNair, unfortunately, I haven't found a solution. I also asked this question in TF repo https://github.com/tensorflow/tensorflow/issues/33162 I hope they will have this new feature or someone could have other simple solutions. – davislf2 Oct 16 '19 at 07:39

1 Answers1

0

In order to load SavedModel to continue training, you can use tf.saved_model.loader.load like follows:

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
  tf.saved_model.loader.load(sess, [tag_constants.SERVING], saved_model_location)

In order to feed new input data, you can get the input tensor names like follows:

signature_def = meta_graph_def.signature_def[tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
inputs = [v.name for v in signature_def.inputs.values()]
input_tensors = [node.split(":")[0] for node in inputs]

then you can make some kinds of feed_dict to feed new inputs to the input tensors. Getting output tensors can be done similarly to the method I outlined above.

Minh Nguyen
  • 755
  • 5
  • 11