Loss becomes NaN in training

Question

I am working on a Faster-RCNN model with below codes. I am using roboflow chess pieces dataset

def get_model(n_classes):
    model = models.detection.fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = models.detection.faster_rcnn.FastRCNNPredictor(in_features, n_classes)
    return model

Dataset class, _getitem_ part


    def __getitem__(self, index):
        id = self.ids[index]
        image = self._load_image(id)
        # target = self._load_target(id)
        target = copy.deepcopy(self._load_target(id))
        boxes = torch.tensor([t["bbox"] for t in target])
        new_boxes = torch.add(boxes[:,:2],boxes[:,2:])
        boxes = torch.cat((boxes[:,:2],new_boxes),1)
        labels = torch.tensor([t["category_id"] for t in target], dtype=torch.int64)
        
        image = torch.from_numpy(image).permute(2,0,1)

        targ = {} 
        targ['boxes'] = boxes
        targ['labels'] = labels
        targ['image_id'] = torch.tensor(index)
        targ['area'] = (boxes[:,2]-boxes[:,0]) * (boxes[:,3]-boxes[:,1]) # we have a different area
        targ['iscrowd'] = torch.tensor([t["iscrowd"] for t in target], dtype=torch.int64)

        return image, targ

The pipeline with above codes works fine without transforms. Predicted bbox seems good and mAPs are between 0.4 and 0.8 after 10 epochs.

However, when I try to implement augmentation like below on above code pieces

def get_transforms(train=False):
    if train:
        transform = A.Compose([
            ToTensorV2()
        ], bbox_params=A.BboxParams(format='pascal_voc',label_fields=["labels"]))
    else:
        transform = A.Compose([
            ToTensorV2()
        ], bbox_params=A.BboxParams(format='pascal_voc',label_fields=["labels"]))
    return transform

Dataset class, _getitem_ part

    def __getitem__(self, index):
        id = self.ids[index]
        image = self._load_image(id)
        # target = self._load_target(id)
        target = copy.deepcopy(self._load_target(id))
        boxes = torch.tensor([t["bbox"] for t in target])
        new_boxes = torch.add(boxes[:,:2],boxes[:,2:])
        boxes = torch.cat((boxes[:,:2],new_boxes),1)
        labels = torch.tensor([t["category_id"] for t in target], dtype=torch.int64)
        
        if self.transforms is not None:
            transformed = self.transforms(image=image, bboxes=boxes, labels=labels)
            image = transformed['image']
            boxes = torch.tensor(transformed['bboxes']).view(len(transformed["bboxes"]),4)
            labels = torch.tensor(transformed["labels"],dtype=torch.int64)

        else:
            image = torch.from_numpy(image).permute(2,0,1)

        targ = {}
        targ['boxes'] = boxes
        targ['labels'] = labels
        targ['image_id'] = torch.tensor(index)
        targ['area'] = (boxes[:,2]-boxes[:,0]) * (boxes[:,3]-boxes[:,1]) # we have a different area
        targ['iscrowd'] = torch.tensor([t["iscrowd"] for t in target], dtype=torch.int64)

        return image, targ

I end up with NaN loss.

This is the last output I get with batch_size 10

Epoch: [0]  [10/18]  eta: 0:02:41  lr: 0.003237  loss: 2.3237 (2.6498)  loss_classifier: 1.4347 (1.8002)  loss_box_reg: 0.7538 (0.7682)  loss_objectness: 0.0441 (0.0595)  loss_rpn_box_reg: 0.0221 (0.0220)  time: 20.2499  data: 0.1298
Loss is nan, stopping training
{'loss_classifier': tensor(nan, grad_fn=<NllLossBackward0>), 'loss_box_reg': tensor(nan, grad_fn=<DivBackward0>), 'loss_objectness': tensor(nan, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>), 'loss_rpn_box_reg': tensor(nan, dtype=torch.float64, grad_fn=<DivBackward0>)}

Why does the loss become NaN?
How to find the problem cause it?

Edit: I am using patches and some of my training examples are empty(no object). Meanwhile the model train on the these patches I noticed the values next to loss values in parenthesis increase.I couldn't find what these parenthesis refer to, but I think it connected with last image or batch.(I was using batch_size 1).

There is a few lines of output while it was process on empty images. I tried with Adam and SGD, results are same.

Epoch: [0]  [17/26]  eta: 0:00:14  lr: 0.003601  loss: 2.4854 (3.9266)  loss_classifier: 1.1224 (2.2893)  loss_box_reg: 0.7182 (1.2226)  loss_objectness: 0.0497 (0.3413)  loss_rpn_box_reg: 0.0116 (0.0735)  time: 1.6587  data: 0.0102 # before empty image
Epoch: [0]  [18/26]  eta: 0:00:12  lr: 0.003801  loss: 2.8132 (61.1689)  loss_classifier: 1.5675 (28.8652)  loss_box_reg: 0.7563 (29.8348)  loss_objectness: 0.1070 (2.2412)  loss_rpn_box_reg: 0.0145 (0.2278)  time: 1.6240  data: 0.0098 # after empty image

For starters remove all but one of the transforms at a time to pin down which specific transform is causing the issue. — DerekG, Jun 03 '22 at 13:11
@DerekG I converted transform function to the simplest I can but it gives nan loss too. I updated the get_transforms part and output part with I just used — Alican Kartal, Jun 03 '22 at 13:36

score 2 · Answer 1 · answered Jun 03 '22 at 19:46

2

If your loss is NaN that usually means that your gradients are vanishing/exploding. You could check your gradients. Also, as a solution I would try to implement gradient clipping and reducing the learning rate. Normalizing the data would help too.

answered Jun 03 '22 at 19:46

cangozpi

89
1
6

Thank you for your answer. Gradient clipping sounds a good solution for my problem. I will research it. I did normalize image except target but I didnt decrease learning rate under 0.0005 because yet I dont know how to choose a suitable learning rate for a specific problem. – Alican Kartal Jun 04 '22 at 22:09

Loss becomes NaN in training

1 Answers1