0

I've built (in jupyter notebook with Python 3.6) a long ML proof of concept, which, in essence, has 3 parts: load & prepare data; train network; use network.

I would like to be able to re-run it from "train network" without the "cost" of preparing the data again & again (even loading the prepared data from a save file takes a noticeable amount of time).

When I run all cells from the start of the network training (the first cell of which includes a K.clear_session to wipe out any previous network - needed if the architecture changes) it fails as, part way through, there are still variables stored (with the same names) which are part of the old graph.

I can see two simple solutions (but you may be able to advise a better method to tidy up):

  1. loop through all the defined variables (Tensors) in global() and del any which are Tensors (implicitly all part of the old session and graph),
    or (better)
  2. loop through all the tensors defined in the (old) graph del'ing them before del'ing the (old) graph.

I can see K.get_uid but can't see how I can use this info to accomplish what I need.

In the meantime I have to reset and rerun the whole workbook everytime I make adjustments to the network.

Is there a better way?

JeToJedno
  • 1
  • 2
  • I have removed "thanks" - it is unnecessary. Also I have replaced "Any advice?" in your question - makes people react badly. –  Mar 26 '18 at 11:03
  • I am not very familiar with Keras, but assuming it uses TF in a normal way, you might be able to do `tf.reset_default_graph()` to delete the graph. If it does not work, it would help if you post precise error and some example code. – iga Mar 27 '18 at 00:28
  • This: https://stackoverflow.com/questions/39352865/resetting-default-graph-does-not-remove-variables would seem to suggest that all this does is create a new graph object rather than delete the old one and all variables in it. That discossion also has some clues as to how I can identify the Tensors in the old graph. I'll experiment in a few hours. – JeToJedno Mar 28 '18 at 17:06

0 Answers0