Resizing MTKView scales old content before redraw

Question

I'm using a MTKView to draw Metal content. It's configured as follows:

mtkView = MTKView(frame: self.view.frame, device: device)
mtkView.colorPixelFormat = .bgra8Unorm
mtkView.delegate = self
mtkView.sampleCount = 4
mtkView.isPaused = true
mtkView.enableSetNeedsDisplay = true

setFrameSize is overriden to trigger a redisplay.

Whenever the view resizes it scales its old content before it redraws everything. This gives a jittering feeling.

I tried setting the contentGravity property of the MTKView's layer to a non-resizing value, but that totally messes up the scale and position of the content. It seems MTKView doesn't want me to fiddle with that parameter.

How can I make sure that during a resize the content is always properly redrawn?

Does setting `layerContentsRedrawPolicy` to `NSViewLayerContentsRedrawDuringViewResize` (`.duringViewResize` in Swift) help? — Ken Thomases, Jul 29 '17 at 00:16
No, I tried a few other options as well, but there is no difference. — Remco Poelstra, Jul 29 '17 at 05:49
How have you configured the `MTKView`? For example, what are the settings for the `paused`, `enableSetNeedsDisplay`, and `autoResizeDrawable` properties? — Ken Thomases, Jul 29 '17 at 11:21
I edited the question to include to include the setup code of the view. — Remco Poelstra, Jul 29 '17 at 20:28
Something else to try: set `presentsWithTransaction` to true. If that isn't sufficient, you may need to follow the advice in the last paragraph of the [docs](https://developer.apple.com/documentation/metalkit/mtkview/1535947-presentswithtransaction) for that property. The issue is that Metal drawing is asynchronous. You commit a command buffer. It is actually scheduled some time later. If you use its `present(_:)`, then it will call the drawable's `present()` method at that time. Even that is delayed. It will wait until all rendering to its texture is completed (not just scheduled). — Ken Thomases, Jul 29 '17 at 21:57
I tried both the commandBuffer's `present` as well as the suggestion from the docs with the drawable's `present`. Both still cause the view the scale the content before the new data is displayed. — Remco Poelstra, Jul 30 '17 at 08:46
With the commandBuffer's `present` though, the scaled version remains on display until after 5 seconds when the following warning is logged: ----CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces, or set CA_ASSERT_MAIN_THREAD_TRANSACTIONS=1 to abort when an implicit transaction isn't created on a main thread.---- I checked that the draw method is called from the main thread. After this warning, the display is suddenly updated to the correct image. — Remco Poelstra, Jul 30 '17 at 08:47
I think I understand the warning, as the commandBuffer's `commit` runs on a different thread. — Remco Poelstra, Jul 30 '17 at 08:56
Just to clarify, did you try the combination of setting `presentsWithTransaction` to true and, in the draw method, after committing the command buffer, doing `waitUntilScheduled()` and then calling the drawable's `present()`? Also, can you confirm that your draw method is being called *during* the resize, hopefully multiple times? — Ken Thomases, Jul 30 '17 at 12:44
Yes, I did those commands in that order. To answer your last question, I installed a runloopObserver in `setFrameSize` and that revealed that the draw method gets called (mostly) after the runloop ends (after the `beforeWaiting` runloop activity). When I explicitly call the draw method from `setFrameSize` it actually works! I'm not sue though whether that's a good place to call draw. I'm also afraid that the `waitUntilScheduled()` call has a performance penalty. Is that true? — Remco Poelstra, Jul 31 '17 at 16:14
Funny enough. This new trick works great for 'normal' view resizing, but when going to fullscreen it won't update the view at all. Not even when both `draw()` and `setNeedsDisplay()` are called from `setFrameSize()`. It will only display the new image data when only `setNeedsDisplay()` is called.... — Remco Poelstra, Jul 31 '17 at 16:35
Well, I'm not sure what to make of those results. Weird. Yes, the `waitUntilScheduled()` call will have a performance penalty. You'd only want to do it during the resize. In that case, it would theoretically reduce how quickly the window/view could update in response to the resize to how fast you could draw frames, which is basically a requirement implicit in your question. — Ken Thomases, Jul 31 '17 at 18:01
Well, it works for the more general case, which is a big win. Thanks for your support! I'm happy to give you the credits of an accepted answer if you have time to write one. — Remco Poelstra, Aug 01 '17 at 06:58

score 8 · Accepted Answer · edited Jul 08 '20 at 17:33

In my usage of Metal and MTKView, I tried various combinations of presentsWithTransaction and waitUntilScheduled without success. I still experienced occasional frames of stretched content in between frames of properly rendered content during live resize.

Finally, I dropped MTKView altogether and made my own NSView subclass that uses CAMetalLayer and resize looks good now (without any use of presentsWithTransaction or waitUntilScheduled). One key bit is that I needed to set the layer's autoresizingMask to get the displayLayer method to be called every frame during window resize.

Here's the header file:

#import <Cocoa/Cocoa.h>
    
@interface MyMTLView : NSView<CALayerDelegate>    
@end

Here's the implementation:

#import <QuartzCore/CAMetalLayer.h>
#import <Metal/Metal.h>

@implementation MyMTLView

- (id)initWithFrame:(NSRect)frame
{
    if (!(self = [super initWithFrame:frame])) {
        return self;
    }

    // We want to be backed by a CAMetalLayer.
    self.wantsLayer = YES;

    // We want to redraw the layer during live window resize.
    self.layerContentsRedrawPolicy = NSViewLayerContentsRedrawDuringViewResize;

    // Not strictly necessary, but in case something goes wrong with live window
    // resize, this layer placement makes it more obvious what's going wrong.
    self.layerContentsPlacement = NSViewLayerContentsPlacementTopLeft;

    return self;
}

- (CALayer*)makeBackingLayer
{
    CAMetalLayer* metalLayer = [CAMetalLayer layer];
    metalLayer.device = MTLCreateSystemDefaultDevice();
    metalLayer.delegate = self;

    // *Both* of these properties are crucial to getting displayLayer to be
    // called during live window resize.
    metalLayer.autoresizingMask = kCALayerHeightSizable | kCALayerWidthSizable;
    metalLayer.needsDisplayOnBoundsChange = YES;

    return metalLayer;
}

- (CAMetalLayer*)metalLayer
{
    return (CAMetalLayer*)self.layer;
}

- (void)setFrameSize:(NSSize)newSize
{
    [super setFrameSize:newSize];

    self.metalLayer.drawableSize = newSize;
}

- (void)displayLayer:(CALayer*)layer
{
    // Do drawing with Metal.
}

@end

For reference, I do all my Metal drawing in MTKView's drawRect method.

Unfortunately this didn't work/improve for me. I use the delegate to draw, maybe that's related. — Remco Poelstra, Apr 12 '19 at 06:40
@RemcoPoelstra Hey, I changed my answer completely. I saw some hangs in `setFrameSize` with my previous answer, so I tried something completely different. — Max, Apr 17 '19 at 22:39
This answer reduces the frequency of glitches for me but doesn't eliminate them. However I figured out that combining this with `presentsWithTransaction` and `waitUntilScheduled` works perfectly. I wrote a blog post and posted a working code sample: http://thume.ca/2019/06/19/glitchless-metal-window-resizing/ — Tristan Hume, Jun 20 '19 at 02:47
I've been having position and scaling issues that I just couldn't seem to resolve, turns out setting the layerContentsPlacement to TopLeft was exactly the last piece of the puzzle for me. Thanks for providing a half decent example to compare against. — Tim Kane, May 07 '20 at 17:33

Jonathan Zrake · Answer 2 · 2018-07-13T15:05:09.107

1

I have the same problem with glitches on view resizing. You can even reproduce it in the HelloTriangle example from the Apple's developer site. However the effect is minimized because the triangle is drawn near the middle of the screen, and it's the content closest to the edge of the window, opposite the corner that drags, that is effected worst. The developer notes regarding use of presentsWithTransaction and waitUntilScheduled do not work for me either.

My solution was to add a Metal layer beneath the window.contentView.layer, and to make that layer large enough that it rarely needs to be resized. The reason this works is that, unlike the window.contentView.layer, which sizes itself automatically to the view (in turn maintaining the window size), you have explicit control of the sublayer size. This eliminates the flickering.

edited Jul 13 '18 at 15:05

answered Jul 06 '18 at 22:32

Jonathan Zrake

603
6
9

I've been playing around with your suggestion, but I can't make this to work. How do you momentarily prevent the MTKView from resizing? Or how do you update it to the correct size once the parent has redrawn? – Remco Poelstra Jul 10 '18 at 14:43
Actually, the solution I have settled on is simpler than the one I suggested here. I just choose a generous layer size, extending beyond the typical size of the view. When the view is resized, it just reveals more of the layer, but never tries to resize it. You'll need your metal layer to be a subLayer of the view's layer. – Jonathan Zrake Jul 10 '18 at 19:20
Does that mean that you use a normal view instead of a MTKView? Is the performance acceptable while you draw ‘too much’? – Remco Poelstra Jul 10 '18 at 19:26
I don't bother with the MTKView class, just create a layer and draw into when things change. I don't know what you mean by draw too much. – Jonathan Zrake Jul 10 '18 at 20:05
Sure, no problem. – Jonathan Zrake Jul 13 '18 at 15:05

score 0 · Answer 3 · answered Jul 08 '20 at 04:48

This helped me - https://github.com/trishume/MetalTest

He uses MetalLayer and careful setting of various properties. Everything is pretty smooth even with two side by side in synchronised scroll views with 45megapixel images.

A link to my original problem How do I position an image correctly in MTKView?

Resizing MTKView scales old content before redraw

3 Answers3

Linked