Train neural network with validation dataset in FANN

Question

As some posts suggest, I start using FANN (http://leenissen.dk/fann/index.php) to do neural network stuff. It is clean and easy to understand.

However, to avoid the over-fitting problem, I need to employ an algorithm that considers validation dataset as an auxiliary. (whats is the difference between train, validation and test set, in neural networks?). Interestingly, FANN wrote that it recommends the developer to consider the over-fitting problem (http://leenissen.dk/fann/wp/help/advanced-usage/).

Now the thing is, as far as I can see, FANN does not have any function to support this feature. The training function in FANN does not provide any arguments to pass the validation dataset in, neither. Am I correct? How do FANN users train their neural networks with validation dataset? Thanks for any help.

score 0 · Answer 1 · answered Jun 08 '16 at 19:08

0

there are Training set error and Validation set error. you train on different epochs and batches and then you combine every time the result between the training and the validation.

when the training error is low and the validation error is high means you are doing over fitting. you need to do some experiments and repeat until you the best model fit your data and not over fitting the training set. you might be interested in reading this paper.

Preventing “Overfitting” of Cross-Validation data

answered Jun 08 '16 at 19:08

Feras

834
7
18

Yeah, I can understand the concept. But I was trying to let FANN do the work for me, instead of implementing this process myself. Any idea? – cyfex Jun 09 '16 at 07:21
based on answer in the link below it is built in this framework you have to implement it yourself. [FANN- cross validation](http://fann-general.narkive.com/aqCKIkLS/cross-validation) – Feras Jun 09 '16 at 08:02

score 0 · Answer 2 · answered Jun 09 '16 at 11:57

You have to split your data into a training data set and a cross-validation data set yourself. You can do this by creating separate input files, or by using a built-in function like fann_subset_train_data (ref).

Once you have the two data sets, you train your NN in whatever way pleases you using your training data set. You then obtain your training error by passing the training data to fann_test_data (ref) and you obtain your cross-validation error by passing the cross-validation data to fann_test_data. Note, this function calculates the mean-squared error (MSE).

Note: users will never train their neural networks with cross-validation data sets - the cross-validation data set is only used for testing!

Thanks, it fully answers my question. But consider the time I accept another one. :) Though it's a really good answer. — cyfex, Jun 09 '16 at 17:30

Luis · Accepted Answer · 2016-07-19T20:42:02.157

You can implement this approach, i.e. dataset split, with FANN yourself, but you need to train each epoch separately, using the function fann_train_epoch.

You start with a big dataset, which you then want to split for the different steps. The tricky thing is: You split the dataset only once, and use only the fist part to adjust the weights (training as such).

Say, you want to have already your 2 datasets: Tran and Validation (like in the example you posted). You first need to store them in different files or arrays. Then, you can do the follwing:

struct fann *ann;
struct fann_train_data *dataTrain;
struct fann_train_data *dataVal;

Assuming that you have both datasets in files:

dataTrain = fann_read_train_from_file("./train.data");
dataVal = fann_read_train_from_file("./val.data");

Then, after setting all network parameters, you train and check the error on the second dataset, one epoch at a time. This is something like:

for(i = 1 ; i <= max_epochs ; i++) {
    fann_train_epoch(ann, dataTrain);
    train_error = fann_test_data(ann, dataTrain);
    val_error = fann_test_data(ann, dataVal);
    if ( val_error > last_val_error )
        break;
    last_val_error = val_error;
}

Of course, this condition is too simple and may stop your training loop too early, if the error fluctuate (as it commonly does: look plot below), but you get the general idea on how to use different datasets during training.

By the way, you may want to save these errors to plot them against the training epoch and have a look after the training ended:

score 0 · Answer 4 · answered Feb 16 '17 at 23:37

You should use validation sub set (usually 1/5 of data) for model selection and architecture of the net in general. Test subset (also 1/5 of data) is used for reporting the error. It should be done like that to avoid reporting an error resulted from same data used for architecture design of the net. You can use the rest of data for training but after finding the model you should do error diagnosis plotting learning curves. Doing so you can reduce the data for training in order to generalize better and not over fitting. Same can be done with the number of nodes and hidden layers.

Train neural network with validation dataset in FANN

4 Answers4