Why does my training loss have regular spikes?

Question

I'm training the Keras object detection model linked at the bottom of this question, although I believe my problem has to do neither with Keras nor with the specific model I'm trying to train (SSD), but rather with the way the data is passed to the model during training.

Here is my problem (see image below): My training loss is decreasing overall, but it shows sharp regular spikes:

The unit on the x-axis is not training epochs, but tens of training steps. The spikes occur precisely once every 1390 training steps, which is exactly the number of training steps for one full pass over my training dataset.

The fact that the spikes always occur after each full pass over the training dataset makes me suspect that the problem is not with the model itself, but with the data it is being fed during the training.

I'm using the batch generator provided in the repository to generate batches during training. I checked the source code of the generator and it does shuffle the training dataset before each pass using sklearn.utils.shuffle.

I'm confused for two reasons:

1. The training dataset is being shuffled before each pass.
2. As you can see in this Jupyter notebook, I'm using the generator's ad-hoc data augmentation features, so the dataset should theoretically never be same for any pass: All the augmentations are random.

I made some test predictions to see if the model is actually learning anything, and it is! The predictions get better over time, but of course the model is learning very slowly since those spikes seem to mess up the gradient every 1390 steps.

Any hints as to what this might be are greatly appreciated! I'm using the exact same Jupyter notebook that is linked above for my training, the only variable I changed is the batch size from 32 to 16. Other than that, the linked notebook contains the exact training process I'm following.

Here is a link to the repository that contains the model:

https://github.com/pierluigiferrari/ssd_keras

Answer 1

I've figured it out myself:

TL;DR:

Make sure your loss magnitude is independent of your mini-batch size.

The long explanation:

In my case the issue was Keras-specific after all.

Maybe the solution to this problem will be useful for someone at some point.

It turns out that Keras divides the loss by the mini-batch size. The important thing to understand here is that it's not the loss function itself that averages over the batch size, but rather the averaging happens somewhere else in the training process.

Why does this matter?

The model I am training, SSD, uses a rather complicated multi-task loss function that does its own averaging (not by the batch size, but by the number of ground truth bounding boxes in the batch). Now if the loss function already divides the loss by some number that is correlated with the batch size, and afterwards Keras divides by the batch size a second time, then all of a sudden the magnitude of the loss value starts to depend on the batch size (to be precise, it becomes inversely proportional to the batch size).

Now usually the number of samples in your dataset is not an integer multiple of the batch size you choose, so the very last mini-batch of an epoch (here I implicitly define an epoch as one full pass over the dataset) will end up containing fewer samples than the batch size. This is what messes up the magnitude of the loss if it depends on the batch size, and in turn messes up the magnitude of gradient. Since I'm using an optimizer with momentum, that messed up gradient continues influencing the gradients of a few subsequent training steps, too.

Once I adjusted the loss function by multiplying the loss by the batch size (thus reverting Keras' subsequent division by the batch size), everything was fine: No more spikes in the loss.

Answer 2

For anyone working in PyTorch, an easy solution which solves this specific problem is to specify in the DataLoader to drop the last batch:

train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=False, 
                                          pin_memory=(torch.cuda.is_available()), 
                                          num_workers=num_workers, drop_last=True)

Answer 3

I would add gradient clipping because this prevents spikes in the gradients to mess up the parameters during training.

Gradient Clipping is a technique to prevent exploding gradients in very deep networks, typically Recurrent Neural Networks.

Most programs allows you to add a gradient clipping parameter to your GD based optimizer.

Answer 4

For me, the solution and cause of the issue were a bit different.

The problem would still occur when defining the batch size to 1 or a proper divisor of the total number of samples (so that the last batch would still be full).

TL;DR;

This is a metric artifact due to averaging that doesn't affect training. The default loss metric is the average loss over the entire current epoch, up to the point where you read it. You can see the per-batch loss by using

def on_batch_begin(batch, logs):
    model.reset_metrics()
    return

lambda_callback = tf.keras.callbacks.LambdaCallback(on_batch_begin=on_batch_begin)

and passing it when training with

model.fit(..., callbacks=[lambda_callback])

Note that this will obviously make the all metrics report only the last training batch loss for each epoch.

Explanation

I managed to pinpoint the problem once I noticed this reset_states() call in the model.fit() method. I also realized that it wouldn't happen over each batch in the batch loop code or train_step, so I went to investigate further.

Looking at the LossesContainer class, it's possible to see that the _total_loss_mean is accumulated with in the __call__() function, and weighted by the batch_size as @Alex mentioned. This value is later then reduced to a mean value (as it's a Mean instance) which finally renders the loss metric we see.

The good news is that _total_loss_mean isn't actually fed into the training process, as it's not the value returned by the LossesContainer's __call__().

The image below shows a training experiment with (gray) and without (pink) the 'fix'. It was done on top of the MNIST Tensorflow example. In it, it' possible to see the averaging effect over the actual data.

Answer 5

I faced a similar problem when using tensorflow. In my case, the issue was not related to mini batch size. The actual problem was that tensorflow wasn't shuffling the data completely due to limitation set by buffer_size as described here https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle For perfect shuffling, the buffer_size should be greater than or equal to the full size of the dataset.