7

I want to use SSIM metric as my loss function for the model I'm working on in tensorflow. SSIM should measure the similarity between my reconstructed output image of my denoising autoencoder and the input uncorrupted image (RGB).

As of what I understood, for using the SSIM metric in tensorflow, the images should be normalized to [0,1] or [0,255] and not [-1,1]. After converting my tensors to [0,1] and implementing SSIM as my loss function, the reconstructed image is black and white instead of a colorful RGB image.

tf.reduce_mean(tf.image.ssim(reconstructed, truth, 1.0))

My model is working fine with MSE (mean squared error), the reconstructed images are colorful (RGB).

using tf.losses.mean_squared_error(truth, reconstructed) the reconstructed image would be RGB image, while using SSIM would give me a one dimensional image.

Why using SSIM as loss function gives me different result than MSE (in terms of reconstructed image channels) in tensorflow?

Yousif H
  • 383
  • 1
  • 2
  • 10
  • May I ask what function did you use for gradient update during these experiments? I am trying to implement SSIM as loss function in tensorflow but it produces NaN values and there is no output image after 1 epoch. The same code was working for mean square losses. Thank you. – shaurov2253 Nov 28 '20 at 05:55
  • @shaurov2253 I just had to specify the “2-nd order optimizer” as in the answer given below (Adam), along with the learning rate and INCREASING the loss function instead of decreasing it (since higher the loss function here (SSIM), the better it is). The rest of the optimization procedure I left to tensoflow to take care of (default values and functions). – Yousif H Nov 29 '20 at 17:32
  • I'm using 'tf.GradientTape()' to calculate the gradient in TF2. Not sure if that's causing problem. – shaurov2253 Dec 03 '20 at 09:12

3 Answers3

12

I was capable of solving the issue by changing the dynamic range of the images to 2.0, since I have images scaled between [-1, 1] by:

loss_rec = tf.reduce_mean(tf.image.ssim(truth, reconstructed, 2.0))

And since a better image quality is shown by a higher SSIM value, I had to minimize the negative of my loss function (SSIM) to optimize my model:

optimizer = tf.train.AdamOptimizer(learning_rate).minimize(-1 * loss_rec)

Yousif H
  • 383
  • 1
  • 2
  • 10
  • 1
    I checked the SSIM by this function for two images in case of unnormalized with maxval=255, normalized between 0 and 1 with maxval=1, and normalized between -1 and 1 with maxval=2. The first two cases give similar result while the last one doesn't. This made me think perhaps this is not the solution. – shaurov2253 Nov 25 '20 at 07:49
  • Excellent idea ! It works. – Thư Sinh Oct 04 '21 at 10:22
  • Instead of calling optmizer.minimize(...), you could also set your `loss_rec = 1 - tf.reduce_mean(tf.image.ssim(truth, reconstructed, 2.))`. It works because SSIM yields a value in [0, 1], with 1 being 100% similar. As it is a loss (the "distance" to being 100% similar), you can do it as `loss = 1 - ssim` – fegemo Feb 15 '23 at 15:42
3

SSIM is designed to only measure the difference between two luminance signals. The RGB images are converted to greyscale before measuring similarity. If that was fed back into the loss function, it wouldn't know if the image was losing color saturation because it wouldn't show up in the error metric. That's just a theory.

James Stanard
  • 31
  • 2
1

The TensorFlow documentation says that no colorspace conversion is applied.

https://www.tensorflow.org/api_docs/python/tf/image/ssim

"Note: The true SSIM is only defined on grayscale. This function does not perform any colorspace transform. (If input is already YUV, then it will compute YUV SSIM average.)"

ai2ys
  • 1,151
  • 10
  • 21
  • If it does not change the colorspace, how does it calculate the SSIM. – MMK Jun 07 '23 at 08:28
  • The source code comment says the following `# Compute average over color channels.`. So there is no conversion, the SSIM get calculated per channel and afterwards these get averaged. – ai2ys Jun 11 '23 at 18:08