Fully connected neural network with constant loss

Question

I am working on a project to predict soccer player values from a set of inputs. The data consists of about 19,000 rows and 8 columns (7 columns for input and 1 column for the target) all of numerical values.

I am using a fully connected Neural Network for the prediction but the problem is the loss is not decreasing as it should.

The loss is very large (1e+13) and doesn’t decrease as it should, it just fluctuates.

This is the function I am using to run the model:

def gradient_descent(model, learning_rate, num_epochs, data_loader, criterion):
    losses = []
    optimizer = torch.optim.Adam(model.parameters())
    for epoch in range(num_epochs):  # one epoch
        for inputs, outputs in data_loader:  # one iteration
            
            inputs, outputs = inputs.to(torch.float32), outputs.to(torch.float32)
            logits = model(inputs)
            loss = criterion(torch.squeeze(logits), outputs)  # forward-pass

            optimizer.zero_grad()  # zero out the gradients
            loss.backward()  # compute the gradients (backward-pass)
            optimizer.step()  # take one step

            losses.append(loss.item())

        loss = sum(losses[-len(data_loader):]) / len(data_loader)
        print(f'Epoch #{epoch}: Loss={loss:.3e}')
    return losses

The model is fully connected neural network with 4 hidden layers, each with 7 neurons. input layer has 7 neurons and output has 1. I am using MSE for loss function. I tried changing the learning rate but it is still bad.

What could be the reason behind this? Thank you!

score 0 · Answer 1 · answered Dec 12 '21 at 06:44

It is difficult to diagnose your problem from the information you provided, but I'll try to point you in some useful directions.

Data Normalization:
The way we initialize the weights in deep NN has a significant effect on the training process. See, e.g.:
He, K., Zhang, X., Ren, S. and Sun, J., Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (ICCV 2015).

Most initialization methods assume the inputs have zero mean and unit variance (or similar statistics). If your inputs violate these assumptions, you will find it difficult to train. See, e.g., this post.

Normalize the Targets:
You are trying to solve a regression problem (MSE loss), it might be the case that your targets are poorly scaled and causing very large loss values. Try and normalize the targets to span a more compact range.

Learning Rate:
Try and adjust your learning rate: both increasing it and decreasing it by orders of magnitude.

Fully connected neural network with constant loss

1 Answers1