0

I am training an image super-resolution CNN for some medical data. I split our dataset to 300 patients for training and 50 patients for testing.

I am using dropout of 50% and aware that dropout can cause similar phenomenon. However, I am not talking about training loss over the training phase and testing loss over the testing phase.

The two metrics are all produced in the testing phase. I am using the testing mode to predict BOTH training patients and testing patients. The result is intriguing.

In terms of the loss of super-resolved image between ground truth, the testing patients are lower than training patients.

The plot below shows that both losses for the training set and the testing set are steadily dropping over epochs. But the testing loss is always lowering than the training. It is very puzzling to me to explain why. Also, the glitch between epoch 60 and epoch 80 is weird to me, too. If someone has an explanation for those problems, it will be more than appreciated. loss chart for both training and testing set over epoch EDIT: The loss are calculate by the root mean square error between predicted output and Ground Truth The CNN take X-Y pair to train, for example the training loss in the above plot are calculated by

def rmse(x, y):
    p = model.predict(x)
    return numpy.sqrt(skimage.measure.compare_mse(p, y))
kaltu
  • 330
  • 5
  • 17
  • What is your loss function? A bit of code would be extremely helpful. Are we talking about classfication (how many classes) or regression. if regression did you do output normalisation, what is the order of magnitude of the output.... – CAFEBABE Oct 23 '18 at 05:29
  • @CAFEBABE the loss in the model is `mse`, I am using cycleGAN pytorch implementation to translate degraded medical image to high resolution one. So the input and output are all 8bit unsigned int 2D array. I'm not very sure is it a classification or regression problem. – kaltu Oct 23 '18 at 06:52
  • Have you looked at the actual data, which is outputted? Does the output make sense or is it some implementation bug? If not it might be that your validation set is an easy subset of your training set ( I have actually seen that in practice). Do you use augmentation for your training set? – Thomas Pinetz Oct 23 '18 at 12:39
  • @ThomasPinetz The generated outputs are all visually make sense. I will check if I can upload some of them to here. It may be the case that the testing set are simply easier than training set. Because we split then by sequence, if the patient id > 300 then it is in the test set, vice versa. The training set is augmented(zoom, random cropping, flip, etc.) automatically in the implementation of cycleGAN. So the bare dataset are not augmented and we are not applying augmentation in testing phase. – kaltu Oct 23 '18 at 12:56
  • Then it seems like your test set is just plain simpler than your training set and therefore your performance on your test set is better. That is not a bad result. – Thomas Pinetz Oct 23 '18 at 13:13
  • I guess it is really like test set being simpler. But just to be sure: you don't feed by accident a bunch of images and you neither have different resolutions between training and test. (I would call it a regression problem, but that you use cycle gan is the relevant piece of information I was looking for). – CAFEBABE Oct 23 '18 at 17:33

0 Answers0