1

I have a neural network for classification with 1 binary output class. The training dataset is unbalanced. 97% for 0 or False, 3% for 1 or True.

I'm using BCELoss for the loss function. What do I send in the BCELoss(weight= ???) argument? How do calculate it? I have tried sending a tensor of (.97, .03) but is generates an error message that there is a mismatch in the number of values being sent. Here is the error message:"RuntimeError: The size of tensor a (16) must match the size of tensor b (2) at non-singleton dimension 0" 16 is the batchsize, 2 is for (.97, 03). I think I need to send a single value times the batchsize?

To be clear, what do I send in the nn.BCELoss(weight = ? ) when the NN has 1 binary output class? How do I cacluate it? THANK YOU!

Thank you for reading this and offering suggestions!!!

John David Deatherage
  • 47
  • 3

2 Answers2

0

To set the class weights for a single node binary output, you pass the weight just for the positive class. For example:

# define the weight for the positive class
pos_weight = torch.tensor([2.0])

# define the BCELoss with weight
criterion = nn.BCELoss(pos_weight=pos_weight)

To calculate the class weight, refer to the following example from here: https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#examine_the_class_label_imbalance

# Scaling by total/2 helps keep the loss to a similar magnitude.
# The sum of the weights of all examples stays the same.
weight_for_0 = (1 / neg) * (total / 2.0)
weight_for_1 = (1 / pos) * (total / 2.0)

class_weight = {0: weight_for_0, 1: weight_for_1}

print('Weight for class 0: {:.2f}'.format(weight_for_0))
print('Weight for class 1: {:.2f}'.format(weight_for_1))
Jesse Sealand
  • 302
  • 1
  • 11
0

https://github.com/ZouJiu1/CNN_numpy/blob/master/net/loss.py#L36-L66

the W can calculated by this way

pos = max(pos, neg) / pos
neg = max(pos, neg) / neg

if the number is too large, you can multiply a small number

pos = 0.1 * max(pos, neg) / pos
neg = 0.1 * max(pos, neg) / neg

or you can clamp it when some value is too big.

pos = clamp(max(pos, neg) / pos, min = 0.3, max = 10)
neg = clamp(max(pos, neg) / neg, min = 0.3, max = 10)

now assume pos = 90, neg = 10 then

pos = 90 / 90
neg = 90 / 10

and you should use nn.CrossEntropyLoss， the W dimension is Class_number. the truth label should be one_hot encoding.

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropy#torch.nn.CrossEntropyLoss

import torch
from torch import nn

W = torch.tensor([90/ 90, 90 / 10], dtype=torch.float32)
loss = nn.CrossEntropyLoss(weight=W)
input = torch.randn(10, 2, requires_grad=True) # batchsize = 10, class_number = 2
label = torch.zeros_like(input)
for i in range(10):
    k = 0 if torch.randn(1) > (0.6 - 0.1) else 1
    label[i, k] = 1

l = loss(input, label)
l

finally, you can refer this answer, the W should be pos_W

How to calculate unbalanced weights for BCEWithLogitsLoss in pytorch

Jiu_Zou
  • 463
  • 1
  • 4