What are the cause(s) of input touch/display lag in android?

Question

I have a very simple app that render a square with opengl, input touch are read by the GLSurfaceView and last position is exchanged with the rendering thread using a volatile variable.

What I observe (and as been also very well described in https://www.mail-archive.com/android-developers@googlegroups.com/msg235325.html) is that there is a delay (a lag) between the touch position and the display.

When activating the developer option to show touch position, I see that when moving rapidly:

• the circle tracking the touch position shown by the system is 1 cm delayed compared to my finger position. I assume that this is system / hardware dependent and cannot be corrected by application.

• but the movement rendered in my application is delayed also 1 cm after the debug circle. So in practice the delay is twice what it could be.

When the finger movement slow down, the final position of the drawing stops and current touch position is synchronized. Exactly as shown in this video (link found on discussion above) http://www.youtube.com/watch?v=fWZGshsXDhM

Still I am clueless on what is causing the second delay, I have timed the exchange of the position from the onTouchEvent to the rendering thread and it is only a few ms. Rendering happens with a frequency of 18ms.

I have try to use velocity to shorten the appearance of the delay by predicting the position 18ms latter but it didn't change any delay perception.

Has someone an explanation for this kind of delay ? Is it caused by system layers before geting access to in it in onTouchEvent ? In which case I would see no way to correct it.

There was no specific answer on the google group. Also what bother me is that in the thread it is mentionned that the delay disappear by using a canva instead of OpenGL. Which raise more question that it answers.

Answer 1

I've been looking into the same thing recently. There's a good explanation of the Android graphics stack internals here:

https://source.android.com/devices/graphics/architecture.html

Essentially SurfaceViews on android represent the producer end of a buffer queue. The consumer of those buffers is SurfaceFlinger which handles updating the surfaces being displayed on the next vsync. The key thing is that this is a queue. Doing eglSwapBuffers in fact doesn't block unless the queue is full. That means if your rendering is simple, GLSurfaceView renders the first couple of frames very quickly, filling the buffer queue, and then settles down into the "render every 16ms" steady state.

You can see the number of filled buffers in the queue with systrace. You'll notice the GLSurfaceView typically starts a vsync period with 1 frame in the queue, when your render function finishes this goes up to 2, meaning you will need to wait for 2 vsyncs before it displays (in fact 3, I think, because SurfaceFlinger adds another for composition, but that's the same for everyone).

Canvas is drawn on demand rather than continuously, and as that drawing typically happens at much less than the display update rate the BufferQueue for that window is typically empty so new content is consumed immediately by SurfaceFlinger on the next vsync.

You can register for callbacks on vsync so you never render more than the display refresh rate. That avoids the "fill the queue as fast as possible" effect of GLSurfaceView's continuous rendering. However I'm not sure that guarantees the queue will not fill up as it is possible that SurfaceFlinger will skip a beat, causing your queue to grow anyway. I've got a question on that here:

Minimize Android GLSurfaceView lag

I've done a test where I render on vsync, but skip a few frames every few hundred to guarantee the queue empties properly. In that case the GL-rendered objects do move in lock-step with the "pointer trail" overlay crosshair. The "debug circle" appears to still be slightly ahead - I think that is implemented as a simple layer reposition during SurfaceFlinger's composition of the drawn surfaces which would put it an additional frame ahead of the other rendering.

Answer 2

As far as I can see, this problem has nothing to do with input lag or OpenGL drawing. My guess is that you have a synchronization issue between your threads.

Input events are issued through the Android UI thread, while drawing happens in the application's OpenGL thread. Assuming your test app is done like the (optimized) example you linked, you are setting an x and y variable in the UI thread, and use those in the drawing thread without synchronization. On a multi-core CPU (as in modern phones like the S3), it is likely these treads are running on different cores (because the OGL thread is active all the time). So each core has a cached version of your x and y variables that is not guaranteed to be updated to the other core immediately.

This problem can be solved by making the variables volatile. From the Java 7 specification:

The Java programming language allows threads to access shared variables (§17.1). As a rule, to ensure that shared variables are consistently and reliably updated, a thread should ensure that it has exclusive use of such variables by obtaining a lock that, conventionally, enforces mutual exclusion for those shared variables.

The Java programming language provides a second mechanism, volatile fields, that is more convenient than locking for some purposes.

A field may be declared volatile, in which case the Java Memory Model ensures that all threads see a consistent value for the variable (§17.4).

The last comment in the group you linked also suggests this solution (it's probably the last comment because it worked). It also comments on the problem of using two volatile variables that are always used and updated together (you might sometimes see the effect of the first variable updated but not the second for one frame). The best solution is to use ONE volatile variable for this, or apply a locking mechanism that also assures atomicity.

I have written several OpenGL apps myself and never had this "input lag" problem, always having used volatile variables. Of course you have to expect the display to be one frame behind the input, because you only update the input data once per frame. So your object being one frame behind the debug circle sounds perfectly normal.