0

I want to save my model at the end of each epoch from this link

I did it in this way:

save:

model_dir = "{0}/epoch_{1}/res".format(train_dir, epoch_num)
saver.save(sess, model_dir, global_step=global_step_val)

restore:

 for ep_num in range(num_epochs):
        model_dir = "{0}/epoch_{1}/".format(train_dir, epoch_num)
        model_meta_file_name = [each for each in 
        os.listdir(model_dir) if each.endswith('.meta')][0]
        meta_path="{0}/{1}".format(model_dir, model_meta_file_name)
        saver = tf.train.import_meta_graph(meta_path)
        saver.restore(sess, tf.train.latest_checkpoint(model_dir))

and continue evaluating my model with evaluation data.

but after 7-8 epochs,my evaluating process gets very slow, I think I am saving some extra things, and after some epochs my graph is very big. How do I solve this?

one point: gpu utilization is almost zero during this evaluation process.I think fro some reason after some epochs,my process does noyt run on GPU

creative_sh
  • 69
  • 1
  • 3
  • 14

1 Answers1

0

saver = tf.train.import_meta_graph(meta_path)

This line should only run once. You call it at every epoch, and that adds new ops to the graph, so the evaluation becomes slower.

Siyuan Ren
  • 7,573
  • 6
  • 47
  • 61