How do I position an image correctly in MTKView?

Question

I am trying to implement an image editing view using MTKView and Core Image filters and have the basics working and can see the filter applied in realtime. However the image is not positioned correctly in the view - can someone point me in the right direction for what needs to be done to get the image to render correctly in the view. It needs to fit the view and retain its original aspect ratio.

Here is the metal draw function - and the empty drawableSizeWillChange!? - go figure. its probably also worth mentioning that the MTKView is a subview of another view in a ScrollView and can be resized by the user. It's not clear to me how Metals handles resizing the view but it seems that doesn't come for free.

I am also trying to call the draw() function from a background thread and this appears to sort of work. I can see the filter effects as they are applied to the image using a slider. As I understand it this should be possible.

It also seems that the coordinate space for rendering is in the images coordinate space - so if the image is smaller than the MTKView then to position the image in the centre the X and Y coordinates will be negative.

When the view is resized then everything gets crazy with the image suddenly becoming way too big and parts of the background not being cleared.

Also when resting the view the image gets stretched rather than redrawing smoothly.

func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    
}


public func draw(in view: MTKView) {
    if let ciImage = self.ciImage  {
        if let currentDrawable = view.currentDrawable {              // 1
            let commandBuffer = commandQueue.makeCommandBuffer()
            
            let inputImage = ciImage     // 2
            exposureFilter.setValue(inputImage, forKey: kCIInputImageKey)
            exposureFilter.setValue(ev, forKey: kCIInputEVKey)
            
            context.render(exposureFilter.outputImage!,                      
                           to: currentDrawable.texture,
                           commandBuffer: commandBuffer,
                           bounds: CGRect(origin: .zero, size: view.drawableSize),
                           colorSpace: colorSpace)
            
            commandBuffer?.present(currentDrawable)                   
            commandBuffer?.commit()
        }
    }
}

As you can see the image is on the bottom left

Answer 1

Thanks to Tristan Hume's MetalTest2 I now have it working nicely in two synchronised scrollViews. The basics are in the subclass below - the renderer and shaders can be found at Tristan's MetalTest2 project. This class is managed by a viewController and is a subview of the scrollView's documentView. See image of the final result.

//
//  MetalLayerView.swift
//  MetalTest2
//
//  Created by Tristan Hume on 2019-06-19.
//  Copyright © 2019 Tristan Hume. All rights reserved.
//

import Cocoa

// Thanks to https://stackoverflow.com/questions/45375548/resizing-mtkview-scales-old-content-before-redraw
// for the recipe behind this, although I had to add presentsWithTransaction and the wait to make it glitch-free
class ImageMetalView: NSView, CALayerDelegate {
    var renderer : Renderer
    var metalLayer : CAMetalLayer!
    var commandQueue: MTLCommandQueue!
    var sourceTexture: MTLTexture!
    let colorSpace = CGColorSpaceCreateDeviceRGB()
    var context: CIContext!
    var ciMgr: CIManager?
    var showEdits: Bool = false
    
    var ciImage: CIImage? {
        didSet {
            self.metalLayer.setNeedsDisplay()
        }
    }
    @objc dynamic var fileUrl: URL? {
        didSet {
            if let url = fileUrl {
                self.ciImage = CIImage(contentsOf: url)
            }
        }
    }
    
    /// Bind to this property from the viewController to receive notifications of changes to CI filter parameters
    @objc dynamic var adjustmentsChanged: Bool = false {
        didSet {
            self.metalLayer.setNeedsDisplay()
        }
    }
    
    override init(frame: NSRect) {
        let _device = MTLCreateSystemDefaultDevice()!
        renderer = Renderer(pixelFormat: .bgra8Unorm, device: _device)
        self.commandQueue = _device.makeCommandQueue()
        self.context = CIContext()
        self.ciMgr = CIManager(context: self.context)
        super.init(frame: frame)
        
        self.wantsLayer = true
        self.layerContentsRedrawPolicy = .duringViewResize
        
        // This property only matters in the case of a rendering glitch, which shouldn't happen anymore
        // The .topLeft version makes glitches less noticeable for normal UIs,
        // while .scaleAxesIndependently matches what MTKView does and makes them very noticeable
        //        self.layerContentsPlacement = .topLeft
        self.layerContentsPlacement = .scaleAxesIndependently
        
        
    }
    
    required init(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    override func makeBackingLayer() -> CALayer {
        metalLayer = CAMetalLayer()
        metalLayer.pixelFormat = .bgra8Unorm
        metalLayer.device = renderer.device
        metalLayer.delegate = self
        
        // If you're using the strategy of .topLeft placement and not presenting with transaction
        // to just make the glitches less visible instead of eliminating them, it can help to make
        // the background color the same as the background of your app, so the glitch artifacts
        // (solid color bands at the edge of the window) are less visible.
        //        metalLayer.backgroundColor = CGColor(red: 0.0, green: 0.0, blue: 0.0, alpha: 1.0)
        
        metalLayer.allowsNextDrawableTimeout = false
        
        // these properties are crucial to resizing working
        metalLayer.autoresizingMask = CAAutoresizingMask(arrayLiteral: [.layerHeightSizable, .layerWidthSizable])
        metalLayer.needsDisplayOnBoundsChange = true
        metalLayer.presentsWithTransaction = true
        metalLayer.framebufferOnly = false
        return metalLayer
    }
    
    override func setFrameSize(_ newSize: NSSize) {
        super.setFrameSize(newSize)
        self.size = newSize
        renderer.viewportSize.x = UInt32(newSize.width)
        renderer.viewportSize.y = UInt32(newSize.height)
        // the conversion below is necessary for high DPI drawing
        metalLayer.drawableSize = convertToBacking(newSize)
        self.viewDidChangeBackingProperties()
    }
    var size: CGSize = .zero
    // This will hopefully be called if the window moves between monitors of
    // different DPIs but I haven't tested this part
    override func viewDidChangeBackingProperties() {
        guard let window = self.window else { return }
        // This is necessary to render correctly on retina displays with the topLeft placement policy
        metalLayer.contentsScale = window.backingScaleFactor
    }
    
    func display(_ layer: CALayer) {
        
        if let drawable = metalLayer.nextDrawable(),
           let commandBuffer = commandQueue.makeCommandBuffer() {
            
            let passDescriptor = MTLRenderPassDescriptor()
            let colorAttachment = passDescriptor.colorAttachments[0]!
            colorAttachment.texture = drawable.texture
            colorAttachment.loadAction = .clear
            colorAttachment.storeAction = .store
            colorAttachment.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
            
            if let outputImage = self.ciImage {
                
                let xscale = self.size.width / outputImage.extent.width
                let yscale = self.size.height / outputImage.extent.height
                let scale = min(xscale, yscale)
                
                if let scaledImage = self.ciMgr!.scaleTransformFilter(outputImage, scale: scale, aspectRatio: 1),
                   
                   let processed = self.showEdits ? self.ciMgr!.processImage(inputImage: scaledImage) : scaledImage {
                    
                    let x = self.size.width/2 - processed.extent.width/2
                    let y = self.size.height/2 - processed.extent.height/2
                    context.render(processed,
                                   to: drawable.texture,
                                   commandBuffer: commandBuffer,
                                   bounds: CGRect(x:-x, y:-y, width: self.size.width, height:  self.size.height),
                                   colorSpace: colorSpace)
                    }
                
                
                
            } else {
                print("Image is nil")
            }
            commandBuffer.commit()
            commandBuffer.waitUntilScheduled()
            drawable.present()
        }
    }
}

Answer 2

let scaleFilter = CIFilter(name: "CILanczosScaleTransform")

That should help you out. The issue is that your CIImage, wherever it might come from, is not the same size as the view you are rendering it in.

So what you could opt to do is calculate the scale, and apply it as a filter:

    let scaleFilter = CIFilter(name: "CILanczosScaleTransform")
    scaleFilter?.setValue(ciImage, forKey: kCIInputImageKey)
    scaleFilter?.setValue(scale, forKey: kCIInputScaleKey)

This resolves your scale issue; I currently do not know what the most efficient approach would be to actually reposition the image

Further reference: https://nshipster.com/image-resizing/

Answer 3

The problem is your call to context.render — you are calling render with bounds: origin .zero. That’s the lower left.

Placing the drawing in the correct spot is up to you. You need to work out where the right bounds origin should be, based on the image dimensions and your drawable size, and render there. If the size is wrong, you also need to apply a scale transform first.