Which SSIM is correct : skimage.metrics.structural_similarity()?

Question

Stackoverflow comunity,

I'm trying to compute SSIM (Structural SIMilarity) between two bmp images on Python. I've found structural_similarity() function implemented in the skimage python library and the equivalent code from the original MatLab implementation which is hosted here. The implimentation is right bellow:

def structuralSimilarityIndex(ref_image, impaired_image, cs_map=False):

    window = Metric.fSpecialGauss(constant.SSIM_FILTER_SIZE,
                                  constant.SSIM_FILTER_SIGMA)
    C1 = (constant.SSIM_Constant_1 * constant.PIXEL_MAX) ** 2
    C2 = (constant.SSIM_Constant_2 * constant.PIXEL_MAX) ** 2

    mu1 = signal.fftconvolve(window, ref_image, mode='valid')
    mu2 = signal.fftconvolve(window, impaired_image, mode='valid')

    mu1_sq = mu1 * mu1
    mu2_sq = mu2 * mu2
    mu1_mu2 = mu1 * mu2

    sigma1_sq = signal.fftconvolve(
        window, ref_image*ref_image, mode='valid') - mu1_sq
    sigma2_sq = signal.fftconvolve(
        window, impaired_image*impaired_image, mode='valid') - mu2_sq
    sigma12 = signal.fftconvolve(
        window, ref_image*impaired_image, mode='valid') - mu1_mu2

    if cs_map:
        return (((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) * (sigma1_sq + sigma2_sq + C2)), (2.0 * sigma12 + C2) / (sigma1_sq + sigma2_sq + C2))
    else:
        return np.mean(((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) * (sigma1_sq + sigma2_sq + C2)))

I'm reading the images using this piece of code :

ref_image = np.asfarray(Image.open('ref_image.bmp').convert('L'))
impaired_image = np.asfarray(Image.open('impaired_image.bmp').covert('L)

The input images shape and dtype of both ref_image and impaired_image are respectively:

(512, 512) float64

(512, 512) float64

I've tested both using the same condition and same input images as follow:

# Using the above code
structuralSimilarityIndex(ref_image, impaired_image, cs_map=False)

# Using the function imported from skimage.metrics
structural_similarity(ref_image, impaired_image, gaussian_weights=False, use_sample_covariance=False)

the result was so much different, here the results:

The SSIM from the Skimage python library:

SSIM : 0.38135154028457885

The SSIM from the code above:

SSIM : 0.8208087737160036

EDIT:

I've added the reading and calling code

The above Python code was from the signal processing library, which is according to the author, the function attempts to mimic precisely the functionality of ssim.m a MATLAB provided by the author's of SSIM

Update :

I've tested the original code which is writing in MatLab on the same images and the result is as follow :

SSIM : 0.8424

Which is not far from the result of the Python implementation given above.

You’ve found the implementation by the authors of the paper presenting SSIM, and you are wondering if it is correct? Surely that must be the reference. Why not directly post a bug report to the skimage project? — Cris Luengo, Oct 29 '19 at 13:10
The Python implementation is provided by the author himself. — asendjasni, Oct 29 '19 at 13:16
@CrisLuengo there is a lot of things I couldn't understand ins the skimage implementation since the variables and the commands were really confusing. — asendjasni, Oct 29 '19 at 13:26
You have to make sure you are operating on the same ranges of data, but overall there should be good agreement between the skimage and matlab versions. The numbers you cite are so far off that it seems to indicate some other problem. Feel free to file an issue with all data and code used on the skimage issue tracker. Sorry if the commands were confusing: we'd love to learn how to make them easier to use. — Stefan van der Walt, Oct 29 '19 at 21:53
@StefanvanderWalt as someone who does not have much experience with Python, I could not understand the code well and either the code posted in the question, since the nominations are ambiguous. I've edited the post by adding the Python code's source. — asendjasni, Oct 30 '19 at 08:37
Worth noting that the two numbers are off by a factor of 2 down to 4 significant figures and there are a lot factors of 2 in the return statement. — Paul Brodersen, Nov 01 '19 at 16:00
@PaulBrodersen really sorry, I dont understand what you mean. — asendjasni, Nov 02 '19 at 20:14
@StefanvanderWalt I've tested the original code of SSIM which is writing in MatLab and I've updated the question according to that. — asendjasni, Nov 04 '19 at 09:42
I was just trying to aid the debugging process by noting that the results only differ by a factor of 2, which is suspicious given the frequency of factors of two in the code. If I were you, I would look at the two sources and check if there are any missing or superfluous factors of two anywhere. — Paul Brodersen, Nov 04 '19 at 10:47
@PaulBrodersen I've tested the original code in a MatLab environment on the same input images and I updated the post accordingly. After that, I opened an issue on the GitHub repo of the skimage library. — asendjasni, Nov 04 '19 at 11:15

score 9 · Accepted Answer · answered Nov 08 '19 at 13:35

I've opened an issue on the scikit-image Github repository and I got an answer. Here the answer, I've change nothing to the answer and you can find it here:

I think the primary issue here is that the way you computed images from PIL results in floating point images, but ones where the values are in the range [0, 255.0]. skimage will assume a range [-1.0, 1.0] for data_range when the input is floating-point, so you will need to manually specify data_range=255.

Also, see the Notes section of the docstring for recommendations to set gaussian_weights=True, sigma=1.5 to more closely match the Matlab script by Wang et. al. (I think recent Matlab also has its own built-in SSIM implementation, but I haven't tried comparing to that case and don't know if it is exactly the same).

ref_image = np.asfarray(Image.open('avion.bmp').convert('L'))
impaired_image = np.asfarray(Image.open('avion_jpeg_r5.bmp').convert('L'))
structural_similarity(ref_image, impaired_image, multichannel=True, gaussian_weights=True, sigma=1.5, use_sample_covariance=False, data_range=255)

gives 0.8292 when I tried it.

Alternatively you can use skimage.io.imread and rgb2gray to read in the data and convert it to grayscale. In that case, values will have been scaled within [0, 1.0] for you and data_range should be set to 1.0.

from skimage.io import imread
from skimage.color import rgb2gray
ref_image = imread('avion.bmp')
ref_image = rgb2gray(ref_image)
impaired_image = imread('avion_jpeg_r5.bmp')
impaired_image = rgb2gray(impaired_image)

structural_similarity(ref_image, impaired_image, multichannel=True, gaussian_weights=True, sigma=1.5, use_sample_covariance=False, data_range=1.0)

gives 0.8265

I think the small difference between the two cases above is probably due to rgb2gray using a different luminance conversion than PIL's convert method.

Yes, this fixed the problem to me, indeed this is how they call it in the official documentation ```ssim_noise = ssim(img, img_noise, data_range=img_noise.max() - img_noise.min())``` — BestDogeStackoverflow, Jun 24 '22 at 12:23
Thanks. I opened a PR to improve the documentation to warn for this: https://github.com/scikit-image/scikit-image/pull/6595 — Martijn Courteaux, Oct 26 '22 at 14:38

Which SSIM is correct : skimage.metrics.structural_similarity()?

1 Answers1