0

I am training the model but I am facing one issue that every time I run the program means if model trained on 10 epochs its accuracy like 80% but after 10 epochs I re run the program, accuracy got changed like 95% and so on, every time I got changed accuracy.Should i stick with very first time or try it multiple times Is this a sign of overfitting or something else?

import numpy as np
import tensorflow as tf
from imgaug.augmenters import Dropout
from sklearn.model_selection import cross_val_score
from keras.wrappers.scikit_learn import KerasClassifier
data = np.load('subclass_features.npy', allow_pickle=True)

from sklearn.model_selection import train_test_split
training_data = np.asarray([i[0] for i in data])  # select upto second last
train_labels = data[:, -1]  # gives the last column vector
print("Shape of training data", training_data.shape)
print("Labels of training data", train_labels.shape)
data = training_data.astype('float32')
data = data / 255
from tensorflow.keras import utils as np_utils

one_hot_train_labels = np_utils.to_categorical(train_labels)


# train_data1, test_data1, train_labels1, test_labels1 = train_test_split(train_data, one_hot_train_labels,random_state=0, test_size=0.3)
def built_classifier():
    classifier = tf.keras.models.Sequential()
    classifier.add(tf.keras.layers.Dense(64, input_shape=(128,), activation='relu'))
    classifier.add(tf.keras.layers.Dense(4, activation='softmax'))
    classifier.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
    classifier.save('SubClassPredictions.h5')
    return classifier


classifier = KerasClassifier(build_fn=built_classifier, epochs=50, batch_size=32, shuffle=True)
accuracies = cross_val_score(classifier, data, one_hot_train_labels, cv=10)
print("Data Accuracy", accuracies)
print("Mean of accuracy is", accuracies.mean())
Hamza
  • 530
  • 5
  • 27
  • 1
    Does this answer your question : https://stackoverflow.com/q/58241065/10077354 – Suraj Jul 10 '20 at 17:56
  • 1
    Yes , it seems similar but actually i got difference about 10-15% , i am seeking answer about this type pf percentage – Hamza Jul 10 '20 at 18:33
  • 1
    Did you end the session or recompile the model or just call the fit function multiple times without re-initializing the model? – Muntasir Wahed Jul 10 '20 at 19:01
  • I recompile the model every time , and everytime fit function calls . I don't know about session – Hamza Jul 11 '20 at 07:36

2 Answers2

1

It may be due to initializing the random weights during forward propagation or else due to splitting of your data randomly. It gives different training and testing datasets at every run. This error can be resolved by providing seed values for the random number generation.

Read this below link you will get some intuition about seeding

Random seed in python

sri dhurkesh
  • 11
  • 1
  • 1
  • 3
  • If weights are initialized randomly , there could be minimal difference but i got 10 - 15 % difference sometimes . – Hamza Jul 11 '20 at 07:37
  • Sometimes it may be due to train and test splitting. when u run the code at first time it will take random training and testing examples at the given ratio and when you rerun the same code again it will take another set of random training and testing examples at the given ratio. **Better post your code..Most of them would like to help you out** – sri dhurkesh Jul 11 '20 at 09:52
  • I have posted the code , it is my model which accepts features from npy file(not pasted here) – Hamza Jul 12 '20 at 09:22
0

You have to set the seed for different things like numpy, cuda etc.. Then your program gives same result. Use below function

def set_random_seed(random_seed):
    torch.manual_seed(random_seed)
    torch.cuda.manual_seed(random_seed)
    torch.cuda.manual_seed_all(random_seed)  # if use multi-GPU

    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

    np.random.seed(random_seed)
    random.seed(random_seed)

Then call with a seed set_random_seed(1234) This will give you same result no matter on which machine you run. You can change the seed if you want. Different seed results in different results.

Palguna Gopireddy
  • 19
  • 6