Meaning of an Epoch in Neural Networks Training

Question

while I'm reading in how to build ANN in pybrain, they say:

Train the network for some epochs. Usually you would set something like 5 here,
trainer.trainEpochs( 1 )

I looked for what is that mean , then I conclude that we use an epoch of data to update weights, If I choose to train the data with 5 epochs as pybrain advice, the dataset will be divided into 5 subsets, and the wights will update 5 times as maximum.

I'm familiar with online training where the wights are updated after each sample data or feature vector, My question is how to be sure that 5 epochs will be enough to build a model and setting the weights probably? what is the advantage of this way on online training? Also the term "epoch" is used on online training, does it mean one feature vector?

score 103 · Accepted Answer · answered Jul 01 '15 at 09:37

One epoch consists of one full training cycle on the training set. Once every sample in the set is seen, you start again - marking the beginning of the 2nd epoch.

This has nothing to do with batch or online training per se. Batch means that you update once at the end of the epoch (after every sample is seen, i.e. #epoch updates) and online that you update after each sample (#samples * #epoch updates).

You can't be sure if 5 epochs or 500 is enough for convergence since it will vary from data to data. You can stop training when the error converges or gets lower than a certain threshold. This also goes into the territory of preventing overfitting. You can read up on early stopping and cross-validation regarding that.

So, the dataset is divide into epochs and each epoch consists of samples? and each epoch used to build a model? — , Jul 01 '15 at 10:42
Does one epoch consist of also running the gradient descent to find the minimum error? If so, why would running it over and over minimize the error more? — mskw, Oct 09 '17 at 20:13

score 1 · Answer 2 · edited Jun 28 '18 at 17:33

1

sorry for reactivating this thread. im new to neural nets and im investigating the impact of 'mini-batch' training.

so far, as i understand it, an epoch (as runDOSrun is saying) is a through use of all in the TrainingSet (not DataSet. because DataSet = TrainingSet + ValidationSet). in mini batch training, you can sub divide the TrainingSet into small Sets and update weights inside an epoch. 'hopefully' this would make the network 'converge' faster.

some definitions of neural networks are outdated and, i guess, must be redefined.

edited Jun 28 '18 at 17:33

Md. Abu Nafee Ibna Zahid

455
1
6
19

answered Jul 09 '16 at 08:54

Jp Ramoso

21
2

I wouldn't say the definition I gave is outdated but just the standard case. For the sake of comprehensibility I didn't include all exceptions. The most general definition of an epoch would be to call it an iteration cycle in which a set of training samples are trained. – runDOSrun Jul 11 '16 at 07:14
true. i was pertaining on the updating point of epoch. but yup. you gave a good answer to user2162652 – Jp Ramoso Jul 11 '16 at 22:56

score 0 · Answer 3 · answered Aug 08 '21 at 13:29

The number of epochs is a hyperparameter that defines the number times that the learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update the internal model parameters.

Meaning of an Epoch in Neural Networks Training

3 Answers3

Linked