Inconsistent/Different Test Performance/Error After Training Neural Network in Matlab

Question

Keeping all parameters constant, I get different Mean Average Percentage Errors on my test data on retraining the neural network. Why is this so? Aren't all components of the neural network training process deterministic? Sometimes, I see a difference of up to 1% on successive trainings.

The training code is below

netFeb = newfit(trainX', trainY', networkConfigFeb);
netFeb.performFcn = 'mae';
netFeb = trainlm(netFeb, trainX', trainY');

% Index for the testing Data
startingInd = find(trainInd == 0, 1, 'first');
endingInd = startingInd + daysInMonth('Feb') - 1 ;

% Testing Data
testX = X(startingInd:endingInd,:);
testY = dailyPeakLoad(startingInd:endingInd,:);
actualLoadFeb = testY;

% Calculate the Forcast Load and the Mean Absolute Percentage Error
forecastLoadFeb = sim(netFeb, testX'';
errFeb = testY - forecastLoadFeb;
errpct = abs(errFeb)./testY*100;
MAPEFeb = mean(errpct(~isinf(errpct)));

What software/library are you using? The start state or even changing the order of samples presented to the ANN will alter the model it learns. This could be caused by a non-static seed. — , Apr 19 '15 at 11:27
I am using MATLAB/train(). I guess train() randomly selects a subset of the data for training and that is why I get different errors on different occasions. Is that correct? — user2441151, Apr 19 '15 at 12:18
Not an expert on this, but I guess its possible the training starts from randomly selected initial values. — A. Donda, Apr 19 '15 at 14:04
I don't think it will be different training samples - it will train using all of the samples you provide. It is far more likely that train shuffles the samples provided, which will cause different convergences. I think what @A.Donda mentioned is also possible, in that the network weights are initialized randomly - again causing different convergences. Could you post your train/test code? — , Apr 19 '15 at 14:37
@ptay89 `netFeb = newfit(trainX', trainY', networkConfigFeb); netFeb.performFcn = 'mae'; netFeb = trainlm(netFeb, trainX', trainY'); % Index for the testing Data startingInd = find(trainInd == 0, 1, 'first'); endingInd = startingInd + daysInMonth('Feb') - 1 ; % Testing Data testX = X(startingInd:endingInd,:); testY = dailyPeakLoad(startingInd:endingInd,:); actualLoadFeb = testY; % Calculate the Forcast Load and the Mean Absolute Percentage Error forecastLoadFeb = sim(netFeb, testX')'; errFeb = testY - forecastLoadFeb; errpct = abs(errFeb)./testY*100; MAPEFeb = mean(errpct(~isinf(errpct)));` — user2441151, Apr 20 '15 at 02:29
@ptay89 Yeah so in this case, since the user is defining the testing/training data manually, there is no randomization of the training data sets selected — krisdestruction, Apr 20 '15 at 20:07

score 1 · Accepted Answer · edited May 23 '17 at 12:06

As A. Donda hinted, since neural networks initialize their weights randomly, they will generate different networks after training. Thus it will give you different performance. While the training process is deterministic, the initial values are not! You may end up in different local minimums as a result or stop in different places.

If you wish to see why, take a look at Why should weights of Neural Networks be initialized to random numbers?

Edit 1:

Notes

Since the user is defining the testing/training data manually, there is no randomization of the training data sets selected

Inconsistent/Different Test Performance/Error After Training Neural Network in Matlab

1 Answers1