Problem definition:
I am implementing a CNN using Tensorflow. The Input and output are of size samples x 128 x 128 x 1
(grayscale image). In loss function I already have SSIM (0-1) and now my goal is to combine SSIM value with perceptual loss using pre-trained VGG16. I have already consulted following answers link1, link2 but instead of concatenating VGG model at end of the main model I would like to compute feature maps inside loss function at specific layers (e.g. pool1, pool2, pool3) and compute overall MSE. I have defined loss function as following:
Combined Loss function:
def lossfun( yTrue, yPred):
alpha = 0.5
return (1-alpha)*perceptual_loss(yTrue, yPred) + alpha*K.mean(1-tf.image.ssim(yTrue, yPred, 1.0))
and perceptual loss:
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.applications.vgg16 import preprocess_input
model = VGG16()
model = Model(inputs=model.inputs, outputs=model.layers[1].output)
def perceptual_loss(yTrue, yPred):
true = model(preprocess_input(yTrue))
P=Concatenate()([yPred,yPred,yPred])
pred = model(preprocess_input(P))
vggLoss = tf.math.reduce_mean(tf.math.square(true - pred))
return vggLoss
The Error I am running into is followig:
ValueError: Dimensions must be equal, but are 224 and 128 for 'loss_22/conv2d_132_loss/sub' (op: 'Sub') with input shapes: [?,224,224,64], [?,128,128,64].
Error arises due to following reason:
yPred
has size None,128,128,1
, after concatenating it three time and pred = model(preprocess_input(P))
I receive feature map named pred
of size None,128,128,64
. While yTrue has size None
and after true = model(preprocess_input(yTrue))
dimension of true
is None,224,224,64
. This eventually creates dimension incompatibility while computing final vggLoss
.
Question
Since I am new to this task I am not sure if I am approaching the problem in the right manner. Should I create samples of size 224x244
instead of 128x128
in order to avoid this conflict, or is there any other workaround to fix this issue?
Thank you !