Image blur detection for iOS in Objective C

Question

I am trying to determine if an image captured from the camera is iOS is blurry or not. I already check the camera focus before taking the picture, but this seems different that if the image is blurry.

I got this working on Android using Open CV, OpenCV with Laplacian formula to detect image is blur or not in Android

This ends up with,

int soglia = -6118750;
if (maxLap <= soglia) { // blurry

I played with this a bit and decreased to -6718750.

For iOS there seems to be less information on doing this. I saw a couple posts of people trying to use Open CV on iOS for this, but they did not seem successful.

I saw this post using Metal on iOS to do this, https://medium.com/better-programming/blur-detection-via-metal-on-ios-16dd02cb1558

This was in Swift, so I manually converted it line by line to Objective C. I think may code is a correct translation, but not sure if the original code is correct or will work in general on camera captured images?

Basic in my testing it always gives me a result of 2, both for average and variance, how can this be used to detect a blurry image, or any other ideas?

- (BOOL) detectBlur: (CGImageRef)image {
NSLog(@"detectBlur: %@", image);
// Initialize MTL
device = MTLCreateSystemDefaultDevice();
queue = [device newCommandQueue];

// Create a command buffer for the transformation pipeline
id <MTLCommandBuffer> commandBuffer = [queue commandBuffer];
// These are the two built-in shaders we will use
MPSImageLaplacian* laplacian = [[MPSImageLaplacian alloc] initWithDevice: device];
MPSImageStatisticsMeanAndVariance* meanAndVariance = [[MPSImageStatisticsMeanAndVariance alloc] initWithDevice: device];
// Load the captured pixel buffer as a texture
MTKTextureLoader* textureLoader = [[MTKTextureLoader alloc] initWithDevice: device];
id <MTLTexture> sourceTexture = [textureLoader newTextureWithCGImage: image options: nil error: nil];
// Create the destination texture for the laplacian transformation
MTLTextureDescriptor* lapDesc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat: sourceTexture.pixelFormat width: sourceTexture.width height: sourceTexture.height mipmapped: false];
lapDesc.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
id <MTLTexture> lapTex = [device newTextureWithDescriptor: lapDesc];

// Encode this as the first transformation to perform
[laplacian encodeToCommandBuffer: commandBuffer sourceTexture: sourceTexture destinationTexture: lapTex];
// Create the destination texture for storing the variance.
MTLTextureDescriptor* varianceTextureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat: sourceTexture.pixelFormat width: 2 height: 1 mipmapped: false];
varianceTextureDescriptor.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
id <MTLTexture> varianceTexture = [device newTextureWithDescriptor: varianceTextureDescriptor];
// Encode this as the second transformation
[meanAndVariance encodeToCommandBuffer: commandBuffer sourceTexture: lapTex destinationTexture: varianceTexture];
// Run the command buffer on the GPU and wait for the results
[commandBuffer commit];
[commandBuffer waitUntilCompleted];
// The output will be just 2 pixels, one with the mean, the other the variance.
NSMutableData* result = [NSMutableData dataWithLength: 2];
void* resultBytes = result.mutableBytes;
//var result = [Int8](repeatElement(0, count: 2));
MTLRegion region = MTLRegionMake2D(0, 0, 2, 1);
const char* bytes = resultBytes;
NSLog(@"***resultBytes: %d", bytes[0]);
NSLog(@"***resultBytes: %d", bytes[1]);
[varianceTexture getBytes: resultBytes bytesPerRow: 1 * 2 * 4 fromRegion: region mipmapLevel: 0];
NSLog(@"resultBytes: %d", bytes[0]);
NSLog(@"resultBytes: %d", bytes[1]);

int variance = (int)bytes[1];

return variance < 2;
}

Answer 1

Your code implies that you assume a varianceTexture with 4 channels of one byte each. But for your varianceTextureDescriptor you may want to use float values, also due the value range of the variance, see code below. Also, it seems that you want to compare with OpenCV and have comparable values.

Anyway, let's maybe start with the Apple documentation for MPSImageLaplacian:

This filter uses an optimized convolution filter with a 3x3 kernel with the following weights:

In Python one could this do e.g. like:

import cv2
import np
from PIL import Image

img = np.array(Image.open('forrest.jpg'))
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
laplacian_kernel = np.array([[0, 1, 0], [1, -4, 1], [0, 1, 0]])

print(img.dtype)
print(img.shape)

laplacian = cv2.filter2D(img, -1, laplacian_kernel)
print('mean', np.mean(laplacian))
print('variance', np.var(laplacian, axis=(0, 1)))

cv2.imshow('laplacian', laplacian)
key = cv2.waitKey(0)

Please note that we use exactly the values given in Apple's documentation.

Which gives the following output for my test image:

uint8
(4032, 3024)
mean 14.531123203525237
variance 975.6843631756923

MPSImageStatisticsMeanAndVariance

We now want to get the same values with Apple's Metal Performance Shader MPSImageStatisticsMeanAndVariance.

It is useful to convert the input image to a gray image. Then apply the MPSImageLaplacian image kernel.

A byte could also only have values from 0 to 255. So for the resulting mean or variance value we want to have float values. We can specify this independently of the pixel format of the input image. So we should use MTLPixelFormatR32Float as follows:

 MTLTextureDescriptor *varianceTextureDescriptor = [MTLTextureDescriptor
                                                       texture2DDescriptorWithPixelFormat:MTLPixelFormatR32Float
                                                       width:2
                                                       height:1
                                                       mipmapped:NO];

Then we want to interpret 8 bytes from the result texture as two floats. We can do this very nicely with a union. This could look like this:

union {
  float f[2];
  unsigned char bytes[8];
} u1;
MTLRegion region = MTLRegionMake2D(0, 0, 2, 1);
[varianceTexture getBytes:u1.bytes bytesPerRow:2 * 4 fromRegion:region mipmapLevel: 0];

Finally, we need to know that the calculation is done with float values between 0 and 1, which practically means that we want to multiply by 255 or 255*255 for the variance to get it into a comparable range of values:

NSLog(@"mean: %f", u1.f[0] * 255);
NSLog(@"variance: %f", u1.f[1] * 255 * 255);

For the sake of completeness, the entire Objective-C code:

id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> queue = [device newCommandQueue];
id<MTLCommandBuffer> commandBuffer = [queue commandBuffer];

MTKTextureLoader *textureLoader = [[MTKTextureLoader alloc] initWithDevice:device];
id<MTLTexture> sourceTexture = [textureLoader newTextureWithCGImage:image.CGImage options:nil error:nil];


CGColorSpaceRef srcColorSpace = CGColorSpaceCreateDeviceRGB();
CGColorSpaceRef dstColorSpace = CGColorSpaceCreateDeviceGray();
CGColorConversionInfoRef conversionInfo = CGColorConversionInfoCreate(srcColorSpace, dstColorSpace);
MPSImageConversion *conversion = [[MPSImageConversion alloc] initWithDevice:device
                                                                   srcAlpha:MPSAlphaTypeAlphaIsOne
                                                                  destAlpha:MPSAlphaTypeAlphaIsOne
                                                            backgroundColor:nil
                                                             conversionInfo:conversionInfo];
MTLTextureDescriptor *grayTextureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatR16Unorm
                                                                                                 width:sourceTexture.width
                                                                                                height:sourceTexture.height
                                                                                             mipmapped:false];
grayTextureDescriptor.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
id<MTLTexture> grayTexture = [device newTextureWithDescriptor:grayTextureDescriptor];
[conversion encodeToCommandBuffer:commandBuffer sourceTexture:sourceTexture destinationTexture:grayTexture];


MTLTextureDescriptor *textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:grayTexture.pixelFormat
                                                                                             width:sourceTexture.width
                                                                                            height:sourceTexture.height
                                                                                         mipmapped:false];
textureDescriptor.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
id<MTLTexture> texture = [device newTextureWithDescriptor:textureDescriptor];

MPSImageLaplacian *imageKernel = [[MPSImageLaplacian alloc] initWithDevice:device];
[imageKernel encodeToCommandBuffer:commandBuffer sourceTexture:grayTexture destinationTexture:texture];


MPSImageStatisticsMeanAndVariance *meanAndVariance = [[MPSImageStatisticsMeanAndVariance alloc] initWithDevice:device];
MTLTextureDescriptor *varianceTextureDescriptor = [MTLTextureDescriptor
                                                   texture2DDescriptorWithPixelFormat:MTLPixelFormatR32Float
                                                   width:2
                                                   height:1
                                                   mipmapped:NO];
varianceTextureDescriptor.usage = MTLTextureUsageShaderWrite;
id<MTLTexture> varianceTexture = [device newTextureWithDescriptor:varianceTextureDescriptor];
[meanAndVariance encodeToCommandBuffer:commandBuffer sourceTexture:texture destinationTexture:varianceTexture];


[commandBuffer commit];
[commandBuffer waitUntilCompleted];

union {
    float f[2];
    unsigned char bytes[8];
} u;

MTLRegion region = MTLRegionMake2D(0, 0, 2, 1);
[varianceTexture getBytes:u.bytes bytesPerRow:2 * 4 fromRegion:region mipmapLevel: 0];

NSLog(@"mean: %f", u.f[0] * 255);
NSLog(@"variance: %f", u.f[1] * 255 * 255);

The final output gives similar values to the Python program:

mean: 14.528159
variance: 974.630615

The Python code and Objective-C code also computes similar values for other images.

Even if this was not asked directly, it should be noted that the variance value is of course also very dependent on the motif. If you have a series of images with the same motif, then the value is certainly meaningful. To illustrate this, here is a small test with two different motifs that are both sharp, but show clear differences in the variance value:

In the upper area you can see the respective image converted to gray and in the lower area after applying the Laplacian filter. The corresponding median or variance values can be seen in the middle between the images.

Answer 2

Swift version

Some things to keep in mind. Laplacian functions with a kernel size of 3x3 (MPSImageLaplacian) can give some different results depending on the size of the image. If you are comparing like images for consistent results you may want to scale the images to a constant size.

guard let image = uiImage.cgImage, let device = MTLCreateSystemDefaultDevice(), let queue = device.makeCommandQueue(), let commandBuffer = queue.makeCommandBuffer() else {
    return
}

let textureLoader = MTKTextureLoader(device: device)

let sourceColorSpace = CGColorSpaceCreateDeviceRGB()
let destinationColorSpace = CGColorSpaceCreateDeviceGray()
let conversionInfo = CGColorConversionInfo(src: sourceColorSpace, dst: destinationColorSpace)
let conversion = MPSImageConversion(device: device, srcAlpha: MPSAlphaType.alphaIsOne, destAlpha: MPSAlphaType.alphaIsOne, backgroundColor: nil, conversionInfo: conversionInfo)

guard let sourceTexture = try? textureLoader.newTexture(cgImage: image, options: [.origin: MTKTextureLoader.Origin.flippedVertically]) else {
    return
}

let grayscaleTextureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: MTLPixelFormat.r16Snorm, width: sourceTexture.width, height: sourceTexture.height, mipmapped: false)
grayscaleTextureDescriptor.usage = [.shaderWrite, .shaderRead]

guard let grayscaleTexture = device.makeTexture(descriptor: grayscaleTextureDescriptor) else {
    return
}

conversion.encode(commandBuffer: commandBuffer, sourceTexture: sourceTexture, destinationTexture: grayscaleTexture)
let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: grayscaleTexture.pixelFormat, width: sourceTexture.width, height: sourceTexture.height, mipmapped: false)
textureDescriptor.usage = [.shaderWrite, .shaderRead]

guard let texture = device.makeTexture(descriptor: textureDescriptor) else {
    return
}

let laplacian = MPSImageLaplacian(device: device)
laplacian.encode(commandBuffer: commandBuffer, sourceTexture: grayscaleTexture, destinationTexture: texture)
let meanAndVariance = MPSImageStatisticsMeanAndVariance(device: device)
let varianceTextureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: MTLPixelFormat.r32Float, width: 2, height: 1, mipmapped: false)
varianceTextureDescriptor.usage = [.shaderWrite]

guard let varianceTexture = device.makeTexture(descriptor: varianceTextureDescriptor) else {
    return
}

meanAndVariance.encode(commandBuffer: commandBuffer, sourceTexture: texture, destinationTexture: varianceTexture)

commandBuffer.commit()
commandBuffer.waitUntilCompleted()
            
var bytes = [Int8](repeatElement(0, count: 8))
let region = MTLRegionMake2D(0, 0, 2, 1)
varianceTexture.getBytes(&bytes, bytesPerRow: 4 * 2, from: region, mipmapLevel: 0)

var result = [Float32](repeating: 0, count: 2)
memcpy(&result, &bytes, 8)

let mean = Double(result[0] * 255.0)
let variance = Double(result[1] * 255.0 * 255.0)
let standardDeviation = sqrt(variance)

Answer 3

There is a sample project from Apple that shows how to find the sharpest image in a sequence of captured images using the Accelerate framework. This might be useful as a starting point.