1

I am getting jpeg data streamed via peer iOS device. I process and display it using CoreImage and Metal + MTKView in the following way. I receive jpeg data and convert it to CIImage. Next I apply appropriate transform on CIImage and render it to CVPixelBuffer. I then display this CVPixelBuffer to MTKView via passthrough shaders (Metal processing & MTKView does not use ciContext that is used by CIImage part of code). Then I convert the pixel buffer to YUV, do some image processing, and display that data in an MTKView as well. Problem is it sometimes gets too laggy and slow. Are there any bottlenecks in the code below that are caused by using CoreImage and Metal? How to make this pipeline better?

   private func handleImageData(_ data:Data) {

    DispatchQueue.main.async {
        if let image = CIImage(data: data) {
            self.displayImage(image)
        }
     }

    }


   private func displayImage(_ image:CIImage) {
       let transformFilter = CIFilter(name: "CIAffineTransform")!
       transformFilter.setValue(image, forKey: "inputImage")
       transformFilter.setValue(NSValue(cgAffineTransform: self.framesTransform), forKey: "inputTransform")
       let sourceImage = transformFilter.value(forKey: "outputImage") as! CIImage

     //Render CIImage to pixel buffer


     var pixelBuffer:CVPixelBuffer? = nil

     CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, self.ciPixelBufferPool!, &pixelBuffer)

    if let pixBuf = pixelBuffer {
        self.ciContext.render(sourceImage, to: pixBuf)

        self.displayPixelBufferOnMTKView(pixBuf)

       //Convert the pixel buffer to YUV as some filters operate only on YUV
        if let yuvPixelBuffer = self.rgb2yuvFrameRenderer?.copyRenderedPixelBuffer(pixBuf) {

           self.processYUVPixelBuffer(yuvPixelBuffer)

        }

    }
}

Here is Metal shader code:

 // Compute kernel
 kernel void kernelRGBtoYUV(texture2d<half, access::read> inputTexture [[ texture(0) ]],
                       texture2d<half, access::write> textureY [[ texture(1) ]],
                       texture2d<half, access::write> textureCbCr [[ texture(2) ]],
                       constant ColorConversion &colorConv [[ buffer(0) ]],
                       uint2 gid [[thread_position_in_grid]])
 {
    // Make sure we don't read or write outside of the texture
     if ((gid.x >= inputTexture.get_width()) || (gid.y >= inputTexture.get_height())) {
       return;
     }

   float3 inputColor = float3(inputTexture.read(gid).rgb);

   float3 yuv = colorConv.matrix*inputColor + colorConv.offset;

   half4 uv = half4(yuv.gbrr);

   textureY.write(half(yuv.x), gid);

   if (gid.x % 2 == 0 && gid.y % 2 == 0) {
      textureCbCr.write(uv, uint2(gid.x / 2, gid.y / 2));
   }

 }
Deepak Sharma
  • 5,577
  • 7
  • 55
  • 131
  • 1
    Why do you need `CVPixelBuffer`s? For instance, you can render a `CIImage` directly into the view (`view.currentDrawable.texture`). – Frank Rupprecht Aug 06 '19 at 06:34
  • The problem is I need CVPixelBuffer to convert it to YUV420, process YUV data, and display it another MTKView. My question is does it really slow the performance the way I am doing? – Deepak Sharma Aug 06 '19 at 11:48
  • Could you maybe elaborate how you convert the data to YUV420? – Frank Rupprecht Aug 06 '19 at 13:42
  • I use Metal compute shader to convert rgb to yuv420. – Deepak Sharma Aug 07 '19 at 08:45
  • You could do the conversion in a custom `CIColorKernel`, then you don't need to render into a `CVPixelBuffer` in between. – Frank Rupprecht Aug 07 '19 at 10:34
  • I don't know if Metal compute shader can easily be converted to CIColorKernel. Any pointers? – Deepak Sharma Aug 07 '19 at 17:22
  • 1
    The documentation from Apple is _very_ sparse, unfortunately. This post is probably a good starting point: https://medium.com/@shu223/core-image-filters-with-metal-71afd6377f4. If you post your shader code, I might be able to help you with the conversion. – Frank Rupprecht Aug 08 '19 at 07:22
  • I updated the question with Metal shader code. – Deepak Sharma Aug 08 '19 at 08:36
  • Hmm… unfortunately Core Image doesn't support creating two outputs at the same time, so this bi-planar approach wouldn't work. I take it this is the pixel buffer format that you would get from camera capture. Do you really need that specific format, or do you just need to work in the YUV color space? – Frank Rupprecht Aug 08 '19 at 11:29
  • Yes, I get this from camera. I convert it to RGB, compress it as jpeg and send it to peer device via network. The other device then reconverts it to YUV. If there was a way to compress and send YUV420 data to other device, that would be best. But how? – Deepak Sharma Aug 08 '19 at 12:01
  • I know too little about compression, unfortunately. What do you need the YUV buffers for? Maybe there's a way to achieve the same based on the RGB data. – Frank Rupprecht Aug 08 '19 at 12:04
  • Some of my filters directly operate on YUV. But then I would need to rewrite them for RGB. Plus more lines in control flow. – Deepak Sharma Aug 08 '19 at 12:23
  • And yes, I need the grayscale data which is Y-component. This Y texture is directly processed by non-Metal/OpenGL/CoreImage code. I use vImage APIs which uses Accelerate Framework. – Deepak Sharma Aug 08 '19 at 12:24
  • Hmm, difficult. With those constraints I don't see much potential for optimization. Maybe you can somehow compress (zip?) the raw data of the senders YUV pixel buffer and transfer that instead. – Frank Rupprecht Aug 08 '19 at 12:51
  • Could you please look at this question - https://stackoverflow.com/questions/57834241/ciimage-display-mtkview-vs-glkview-performance – Deepak Sharma Sep 07 '19 at 18:55

0 Answers0