2

Using Keras, I am trying to loop a training session 10 times with different splits of data. Though, after every loop my accuracy increases a lot probably because it doesn't reset and sees new data in new groups (data trained on might appear in test next loop)

I expected model.fit to reset it over, as pr. an answer here saying it does so but I can't get it to. I then tried K.clear_session() in the start of the loop, as pr. example 1 here, but it does nothing. I can save an untrained model the first time and reload it at start of loop, but this seems like a bad method/bad practice. What can I do instead/am I doing wrong?

from tensorflow.keras import backend as K

for i in range(0, 10):
    print("Starting loop " + str(i))
    K.clear_session()
    model = keras.Model(inputs=inputs, outputs=outputs, name="SchoolProject")
    model.compile(loss=tensorflow.keras.losses.binary_crossentropy, optimizer=tensorflow.keras.optimizers.Adam(lr=hpInitialLearningRate), metrics=['accuracy'])
    trainData, valData, testData, trainTruth, valTruth, testTruth = getTrainValAndTestSet()
    model.fit(trainData, trainTruth, epochs=hpEpochs, verbose=1, callbacks=callbacks_list, validation_data=(valData, valTruth))
    score = model.evaluate(testData, testTruth, verbose=1)
    print('Test loss:', score[0])
    print('Test accuracy:', score[1])
    testAccList.append(score[1])
    print("Ending loop " + str(i))
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
TDuncker
  • 45
  • 7

1 Answers1

4

The easiest way would be to define your model inside the loop. Here's an example. You'll see that every iteration, the accuracy starts at random before improving.

import tensorflow as tf

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255

for i in range(5):
    model = tf.keras.Sequential([
            tf.keras.Input(shape=(28, 28)),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(32, activation='relu'),
            tf.keras.layers.Dense(64, activation='relu'),
            tf.keras.layers.Dense(10, activation="softmax"),
        ])
    model.compile(loss="sparse_categorical_crossentropy", optimizer="adam",
                  metrics=["accuracy"])
    model.fit(x_train, y_train, batch_size=16, epochs=1, validation_split=0.1)

Resetting the weights manually is a little more complicated.

Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • Thanks. Seems like it works, although is this the proper way to loop it? – TDuncker Oct 30 '20 at 07:57
  • Yes. Why wouldn't it be? – Nicolas Gervais Oct 30 '20 at 11:14
  • I'm running into problems other places, like when using Tensorboard because it interprets everything as one run. The graphs would then show e.g. accuracy slowly increasing and then darting back to 50% and slowly up again, as it does a new "loop". And someone mentioned I should be able to just loop fit and avoid defining a new model every time (but I don't care about this step, as it doesn't take long anyways). The reason I wanted to keep it as part of the loop was because I wanted to calculate some things using all runs done in the loop. – TDuncker Oct 30 '20 at 11:49
  • Well to be frank with you this is a solution to the question you asked. How to implement Tensorboard is out of scope for this question. It is certainly possible to make Tensorboard reset the metrics, though. It's just out of scope for _this_ post. – Nicolas Gervais Oct 30 '20 at 12:10
  • Right, I understand that, but it wasn't a question specifically for tensorboard only. I was just looking for guidance on what is the best "code practice" for looping a network, because I'm not experienced with formatting code properly. I often implement things that technically work and later get told there's a better or "proper" way (for readability and compatability). But this works, and I guess it would be doable to make a second file running a network like this multiple times and then returning the metrics I want. – TDuncker Oct 30 '20 at 12:57