2

I want to process an image with a custom Metal-based convolution CIKernel and blend the result with other images into one composition. But I’m observing strange clipping behavior during blending:

To illustrate the point, I created a simple composition like so:

let inputImage = CIImage(image: inputImage)
let filter = MyCustomFilter()
filter.setValue(inputImage, forKey: kCIInputImageKey)
let filteredImage = filter.outputImage!.transformed(by: .init(translationX: 360, y: 300))
let background = CIImage(color: .white).cropped(to: CGRect(origin: .zero, size: CGSize(width: 1500, height: 1000)))

let composedImage = filteredImage.composited(over: background)

When MyCustomFilter does nothing (return src.sample(src.coord())), the result looks like this: enter image description here

Now, if I perform an actual convolution (sample pixels around the middle pixel) in the filter like so

float4 testKernel(sampler src, destination dest) {
    return src.sample(src.transform(dest.coord() + float2(1,1)));
}

the result suddenly looks like this: enter image description here

As you can see there’s suddenly some clamp-to-extend behavior added to the image when blending, even though its extend is still that of the inputImage.

I observed that Apple’s convolution filters return a larger output image that also contains a halo region depending on the filter parameters. So I played around a lot and found that if I tell the kernel to render into a larger extend and crop it afterwards, the blending again works as expected:

let output = self.kernel.apply(extent: inputExtent.insetBy(dx: -1, dy: -1),
                          roiCallback: roiCallback,
                            arguments: [inputImage])
return output.cropped(to: inputExtent)

It seems the blending filter somehow tries to sample the pixels just outside the image’s extent to determine how to fill the blank space. This should be transparent by default, but seemingly the convolution operation alone adds some clamp-to-edge behavior, even though I don’t change the result extent.

Does anyone know if this is a bug and/or what the best approach would be to combine custom convolution filters with compositions?

Roi Mulia
  • 5,626
  • 11
  • 54
  • 105
Frank Rupprecht
  • 9,191
  • 31
  • 56

0 Answers0