24

There are a few similar questions out there on SO (links at end), but none of them has allowed me to fix my problem, so here goes:

I'm using OpenGL rendering to make an image tiling and caching library for use in a game project, and I want to hijack the physics of the UIScrollView to allow the user to navigate around the images (since it has nice bounce behaviour, might as well use it). So I have a UIScrollView which I'm using to get the rendering view for my textures, but there's a problem - moving around on the scroll view prevents the CADisplayLink from firing until the user has finished scrolling (which looks horrible). One temporary fix has been to use NSRunLoopCommonModes instead of the default run mode, but unfortunately this breaks some aspects of scroll view behaviour on certain phones I'm testing on (the 3GS and simulator seem to work fine, while the iPhone4 and the 3G don't).

Does anyone know how I could get around this clash between the CADisplayLink and the UIScrollView, or know how to fix the UIScrollView working in other run modes? Thanks in advance :)

Promised links to similar questions: UIScrollView broken and halts scrolling with OpenGL rendering (related CADisplayLink, NSRunLoop)

Animation in OpenGL ES view freezes when UIScrollView is dragged on iPhone

Community
  • 1
  • 1
Adam
  • 713
  • 1
  • 8
  • 16

4 Answers4

43

It's possible that slow updates on the main thread triggered by the CADisplayLink are what's breaking UIScrollView's scrolling behavior here. Your OpenGL ES rendering might be taking long enough for each frame to throw off the timing of a UIScrollView when using NSRunLoopCommonModes for the CADisplayLink.

One way around this is to perform your OpenGL ES rendering actions on a background thread by using a Grand Central Dispatch serial queue. I did this in my recent update to Molecules (source code for which can be found at that link), and in testing with using NSRunLoopCommonModes on my CADisplayLink, I don't see any interruption of the native scrolling behavior of a table view that's onscreen at the same time as the rendering.

For this, you can create a GCD serial dispatch queue and use it for all of your rendering updates to a particular OpenGL ES context to avoid two actions writing to the context at the same time. Then, within your CADisplayLink callback you can use code like the following:

if (dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_NOW) != 0)
{
    return;
}

dispatch_async(openGLESContextQueue, ^{

    [EAGLContext setCurrentContext:context];

    // Render here

    dispatch_semaphore_signal(frameRenderingSemaphore);
});

where frameRenderingSemaphore is created earlier as follows:

frameRenderingSemaphore = dispatch_semaphore_create(1);

This code will only add a new frame rendering action onto the queue if one isn't in the middle of executing. That way, the CADisplayLink can fire continuously, but it won't overload the queue with pending rendering actions if a frame takes longer than 1/60th of a second to process.

Again, I tried this on my iPad here and found no disruption to the scrolling action of a table view, just a little slowdown as the OpenGL ES rendering consumed GPU cycles.

Brad Larson
  • 170,088
  • 45
  • 397
  • 571
  • I think you're correct in your guess as to how I was breaking the scroll behaviour, that would explain why it only happened on some devices. Will implement when I get the chance :) – Adam May 11 '11 at 03:54
  • Brad: Great finding! Could you provide any tips for implementing a similar solution for use in iOS 3.x where Grand Central Dispatch is not available? – Ricardo Sanchez-Saez May 14 '11 at 17:02
  • @rsanchez - Unfortunately, there really isn't a good non-GCD solution for this, short of writing a whole mess of code to implement your own lockless queueing system. NSOperationQueue has too much overhead in my testing for something firing at 10+ FPS on the iOS devices. It is for this reason that I am now making all of my applications require 4.0 going forward. Out of hundreds of thousands of users, only three have complained about me dropping 3.x support. Also, I imagine Apple will no longer allow you to target 3.x when 5.0 hits, like they dropped 2.x as a target when 4.0 came out. – Brad Larson May 15 '11 at 16:25
  • Thanks for your thoughts Brad. I'm using the cocos2d-iphone engine, and the CADisplayLink callback was being called 60 times a second when NSRunLoop is in kCFRunLoopDefaultMode, but when you start moving the scrollview and the mode changes to UITrackingRunLoopMode, the callback was being called more often (I got around 90 fps). So many OpenGL calls were disrupting the UIScrollView, as you found out. I finally ended up implementing a simple throttler so OpenGL's draw method isn't being called more than 60 times a second when in UITrackingRunLoopMode. Now the UIScrollView works beautifully! – Ricardo Sanchez-Saez May 15 '11 at 22:47
  • 1
    We plan to target 3.1+ devices, although maybe you are right and we should drop 3.x altogether... – Ricardo Sanchez-Saez May 15 '11 at 22:49
  • Obvious observation: having handed your context off to GCD, it's no longer safe to assume it's usable on your main thread. So this solution works only if — once rendering has started — you don't make any GL calls outside of the render loop. It's probably safer to create a new context in a share group with your existing context and pass that backwards, following the EAGLSharegroup documentation for synchronising new objects between the contexts (not much to it, basically glFlush on each). – Tommy May 16 '11 at 00:26
  • @Tommy - I intended the serial GCD queue to be a lockless means of guaranteeing that only one action was accessing the context at any moment. Of course, any rendering, framebuffer setup, etc. that touches this context would need to be encapsulated in a block to be performed on this serial queue. As I show above, every frame render is placed on this queue, but so are actions that resize FBOs or change out textures in response to orientation changes, as well as responses to user interaction. In practice, I've not once encountered a simultaneous access to the context on any device. – Brad Larson May 16 '11 at 01:54
  • I encounter a similar problem in Sparrow. Your answer helps me fix it! Thanks again ^^ (my question is here http://stackoverflow.com/questions/9512706/sptween-freezes-when-dragging-zooming-a-view-inside-uiscrollview/9600258#9600258) – Hlung Mar 07 '12 at 11:09
  • I'm not using UIScrollView or any other UIKit for that matter. I do both physics and rendering on the main thread's display link callback, and the performance is good (my game is simple). Perhaps a bit fill-rate bound when there are many blended sprites on screen. Do you recommend introducing GCD in this case? I am seriously thinking about decoupling rendering and logic/physics, if anything because my next game might be more taxing, but it comes at a complexity cost... – Nicolas Miari Jun 22 '12 at 07:29
  • 3
    @ranReloaded - By making your rendering occur on a non-main thread, you can see some nice performance improvements due to parallelizing the GPU- and CPU-bound processing (uploading data from the CPU while the GPU is still working on the last set, etc.). This can be of particular help on a multicore system, where I've seen boosts of up to 40% in rendering performance just from backgrounding the frame rendering. Even on single-core machines, I saw a 10-20% improvement. – Brad Larson Jun 22 '12 at 18:00
  • Yes, but I guess it's not as simple as just 'drawing'. Everytime I need to create a sprite of a new kind (= new VBO) or upload a new texture, that must be done in the bg thread as well, and these things are triggered by user input. Far from trivial, it requires a significant overhaul... – Nicolas Miari Jun 22 '12 at 18:56
  • Oh, wait... GL object creation can be done on a separate bg thread with its own context (common sharegroup), except VAOs... – Nicolas Miari Jun 22 '12 at 19:00
  • 1
    @ranReloaded - It's not so bad, if you use a serial dispatch queue. Using a single serial dispatch queue for all actions that touch a particular OpenGL ES context will guarantee that you won't get simultaneous access to that context from multiple threads. In my case, it was easy to migrate to that from running everything on the main thread. Just wrap all actions that touch this context in blocks in dispatch then synchronously or asynchronously to the serial queue as needed. – Brad Larson Jun 22 '12 at 19:10
  • Hey talking about molecules, I just downloaded it. Silky smooth! Off topic, But the near-plane clipping... Did you consider reducing the view angle instead of bringing the model closer when zooming up? I know the perspective is not quite the same, though... – Nicolas Miari Jun 22 '12 at 19:26
  • 1
    I just implemented this solution with a GLKView which is owned by a GLKViewController, and found that I had to call '[(GLKView*)self.view display];' on the main thread at the end of the displayLink callback in order to get everything to update. Many thanks to the contributors on this thread; my opengl world's perspective is now controlled by a UIScrollView, getting me all that nice fluid motion for free. – jankins Jun 29 '12 at 22:04
  • I tried using this technique on my OpenGL ES view that is embedded in a scroll view. The dispatch_semaphore_wait call returns from the run loop method always and it never draws. Added the display link to the NSRunLoopCommonModes. – Proud Member Sep 27 '12 at 17:49
  • In case someone can't get this to work, look here: http://stackoverflow.com/questions/12627581/why-does-dispatch-semaphore-wait-return-yes-all-the-time-even-when-im-not-scr/12628071#12628071 – Proud Member Sep 27 '12 at 18:44
  • Do we know when the display refreshes relative to CADisplayLink? I'm asking because if it is after the CADisplayLink callback returns shouldn't dispatch_sync be used instead of dispatch_async? – Xavier Oct 18 '12 at 21:38
  • 1
    @Xavier - That's a good question, and one that I've experimented with myself. In my experience, CADisplayLink seems to fire a little before the screen refreshes, so you have a few milliseconds to render before you get kicked into the next refresh. Still, a synchronous dispatch has severe downsides in that it blocks the main queue and breaks rendering, like is the problem in this question. Asynchronous dispatches for OpenGL ES rendering also nicely parallelize data upload and rendering, and can lead to huge performance wins on multicore devices. I've tested both approaches, and async wins here. – Brad Larson Oct 18 '12 at 22:32
  • Using this solution, I find that the rendering is very rarely skipped, but my tableview does very often lock up - like it used to before implementing this. The table view will get 'stuck' where cells are unresponsive, the scrollbar doesn't fade, and if in a position where it should bounce, no bounce occurs. I'm at a complete loss of how to continue. – Doc Jul 19 '13 at 16:00
  • @Doc - While offloading your rendering to a synchronous queue and using a semaphore will allow for scrolling that doesn't break or rendering that doesn't pause, you still have limited GPU resources on a device. If you're chewing all of this up in your OpenGL ES rendering, Core Animation performance will start to suffer. You'll either need to optimize your OpenGL ES rendering to reduce GPU load or scale back the framerate to preserve smooth animation in the UI. – Brad Larson Jul 19 '13 at 16:12
  • @BradLarson I've noticed that on my faster devices (iPad 4th gen), I have these lock-up issues more often than my slower devices (iPad 2nd/3rd gen), despite them all using the same fps (20) for openGL rendering. My openGL is being used to draw 2d video, and is fairly efficient to my knowledge, I'm not sure how I can optimize it further. – Doc Jul 19 '13 at 16:17
4

The answer at the following post works very well for me (it appears to be quite similar to Till's answer):

UIScrollView pauses NSTimer until scrolling finishes

To summarize: disable the CADisplayLink or GLKViewController render loop when the UIScrollView appears and start a NSTimer to perform the update/render loop at the desired framerate. When the UIScrollView is dismissed/removed from the view hierarchy, re-enable the displayLink/GLKViewController loop.

In the GLKViewController subclass I use the following code

on appear of UIScrollView:

// disable GLKViewController update/render loop, it will be interrupted
// by the UIScrollView of the MPMediaPicker
self.paused = YES;
updateAndRenderTimer = [NSTimer timerWithTimeInterval:1.0f/60.0f target:self selector:@selector(updateAndRender) userInfo:nil repeats:YES];
[[NSRunLoop mainRunLoop] addTimer:updateAndRenderTimer forMode:NSRunLoopCommonModes];

on dismiss of UIScrollView:

// enable the GLKViewController update/render loop and cancel our own.
// UIScrollView wont interrupt us anymore
self.paused = NO;
[updateAndRenderTimer invalidate];
updateAndRenderTimer = nil;

Simple and effective. I'm not sure if this could cause artifacts/tearing of some sort since the rendering is decoupled from screen refreshes, but using CADisplayLink with NSRunLoopCommonModes totally breaks the UIScrollView in our case. Using NSTimer looks just fine for our app and definitely a whole lot better than no rendering.

Community
  • 1
  • 1
Jeroen Bouma
  • 593
  • 6
  • 16
  • This is honestly the best solution. Constant framerate, doesn't allow UI to block it... Should be selected answer. – Keegan Jay Jun 20 '13 at 22:52
4

My simple solution is to halve the rendering rate when the run loop is in tracking mode. All my UIScrollViews now work smoothly.

Here is the code fragment:

- (void) drawView: (CADisplayLink*) displayLink
{
    if (displayLink != nil) 
    {
        self.tickCounter++;

        if(( [[ NSRunLoop currentRunLoop ] currentMode ] == UITrackingRunLoopMode ) && ( self.tickCounter & 1 ))
        {
            return;
        }

        /*** Rendering code goes here ***/
     }
}
Mindbrix
  • 129
  • 6
  • Doesn't this only helps when there is a touch event? Lets say you scroll but there is momentum left. I thought the tracking mode only restrict when there is a touch event? – mskw Mar 17 '13 at 17:58
  • 1
    Quite the opposite. It helps when there is any tracking or momentum. However, I've deprecated this approach in favour of Brad Larson's GCD solution detailed above, which works superbly. – Mindbrix Mar 18 '13 at 16:24
0

Even though this is not the perfect solution, it still might work as a workaround; You could ignore the display link availability and use NSTimer for updating your GL-layer instead.

Till
  • 27,559
  • 13
  • 88
  • 122
  • The current solution we have is not exactly perfect either (ignoring the display loop altogether and calling render on the didScroll delegate method). The real problem is that we want this class I'm writing to fit into a pre-existing framework for OpenGL games on the iPhone that we have. Ultimately, if the display link can't play nicely with the scroll view we might have to just re-implement the bits of scroll view behaviour we like, but that would take a while :( – Adam May 10 '11 at 02:07
  • 1
    Been there, got the T-shirt ;).... Friction scrolling with bounce-back is not thaaat trivial to implement. – Till May 10 '11 at 14:18