5

I have a few problems (actually more, but these are the more critical ones) with Android's MediaProjection API. Reading the graphics architecture won't really help, so I'm just looking to understand if I skipped something in my code flow.

Let's assume that:

  1. I have a dedicated GL rendering thread, initialized, and with a GL texture generated on it. I set a default buffer size of WxH for the texture.

  2. I create a SurfaceTexture using the GL texture, and a Surface for this surface texture.

  3. Create a virtual display of size WxH via MediaProjection, and set its surface to the above surface.

Problem 1: everything works either perfectly (full frames come in correctly), or not (all frames are black; or only half of each frame is visible for example - same half for all frames; or some portions of the screen are repeated to other portions, sometimes even skewed).

Problem 2: while time is spent in some full-screen GL games, after a FIXED amount of time (about 4 minutes), all incoming frames are frozen (e.g. I receive "new" frames, but its actually one and the same image). Reading that with glReadPixels confirms the results - problem is, the actual display is way past that frame. The only way to force it to "recover" is to bring up the status bar or the navigation bar, which instantly starts sending me correct frames. Ofcourse, after another 4 minutes, it happens again...

Problem 3: calling resize() on the VirtualDisplay, after calling also setDefaultBUfferSize() to the GL texture, ends up in 90% of the cases to exhibit problemm #1 (either black/cut frames, artifacts of other screen areas...)

I am using the sequence of calls to updateTextureImage -> GL texture draw in the same thread, so my normal understanding is that it should never happen that I am somehow reading from a half-full GL buffer, or something... right?

I have tested this problem also by rendering the VirtualDisplay directly to the surface of a MediaCodec (no custom GL involved) - same behaviour. update Actually since MediaCodec has a fixed-size created surface, the bug cannot be reproduced because we can only resize the virtual display, but not the encoder's surface size, so this is not really a bug (but even like this it would be nice for the VirtualDisplay to resize the surface accordingly somehow).

I feel that something is leaking when closing a virtual display, or not initializing correctly between the creations of the VirtualDisplay, because it is not consistent. It may very well happen that a brand new MediaProjection screen capture permission, with a brand new virtual display, with a just-created surface texture, will end up giving me only half-cut frames... leaving me with a big poker face...

PS: all of this is happening on a Nexus 6 with Android 6.0.1.

Adrian Crețu
  • 878
  • 8
  • 17
  • 1
    Sounds a little like https://github.com/google/grafika/issues/43, in that it runs for N minutes and then stops updating. The person who filed the bug found a strange workaround. Are you using a single EGL context? Seeing screen parts repeated is usually an artifact of GPU tiling, and often caused by failing to `glClear()` the screen. (I think some bigflake code deliberately skips the `glClear()` because it's rendering on the full screen; I'd recommend clearing the screen to bright red to make any "holes" noticeable.) – fadden Mar 29 '16 at 18:31
  • I have a similar set up as you My problems only seem to happen on screen rotation and when I try to resize the virtual display/surface texture (as described [here](http://stackoverflow.com/questions/35209004/resize-surface-when-resizing-virtualdisplay)). [This](https://code.google.com/p/android/issues/detail?id=197377) is what I see along with some screen "duping" When I first create the VirtualDisplay and Surface/SurfaceTexture there are no problems for me. It doesn't happen 100% of the time on rotation which leads me to believe it is a race condition. I'm using a Nexus 5x on 6.0.1. – EncodedNybble Mar 29 '16 at 20:11
  • @fadden - I have no control over glClear when rendering directly to a MediaCodec surface. In my GL renderer, I do call glClear (for letterboxing purposes, to clear entire viewport when rotating) and then render the texture (which supposedly was updated by the latest updateTextureImage call). The problem is that the texture itself is incomplete.in its own bounds. As EncodedNybble mentions, it also seems to happen more often when the actual display rotation is orthogonal to the virtual display size (when it scales and centers by itself). – Adrian Crețu Mar 29 '16 at 20:35
  • Another thing I noticed is that turning the screen off and back on after a few seconds may fix the problem. I suspect some issue with the texture buffer size, or maybe the producer not using buffers of a correct size, after texture.setDefaultBufferSize was called? Maybe its using old buffers? The docs say that the Camera producer overrides the size, why doesn't the VirtualDisplay do this also? I think nothing is rendered if I don't set the buffer size initially, IIRC. Is there someway to *force* deleting old buffers of a GL texture? – Adrian Crețu Mar 29 '16 at 20:44

1 Answers1

1

I gave up and just recreated the virtual display all-together, using a new Surface of a new SurfaceTexture (but kept the GL texture). This gets rid of the flickering and half-sized frames. I also had some logcat warnings that the BufferQueue of the previously used SurfaceTexture was abandoned, and got 1 or 2 more onFrameAvailable callbacks after calling release() on the VirtualDisplay (it's probably async and should wait for the stop() callback, but it gets too complicated at that point).

And even after doing all this, something very weird still happens: sometimes the capturing will only capture MY APPLICATION. If I swipe down the status bar, or go to Home, everything turns to black (even if my app is behind the status bar window). The only solution after this is to turn the screen off/on, or to render a Camera/Camera2 preview to my SurfaceTexture, and then recreating back the VirtualDisplay. Something is definitely wrong behind the scenes regarding the texture.

It looks like VirtualDisplay.resize() is broken, and the same applies with creating a new Virtual Display using an existing SurfaceTexture.

Adrian Crețu
  • 878
  • 8
  • 17
  • I'm afraid I don't get the same behavior you do here. Even using a new `VirtualDisplay` and a new `Surface` and new `SurfaceTexture` and even a new OpenGL texture (I do get the abandoned warnings) I still get the "gray bar" and strange torn/duplicated/smaller that I mentioned earlier when I try to make a new `VirtualDisplay` after an orientation change. I will agree `VirtualDisplay` at least on these devices on 6.0.1 seems to be broken when dealing with resizes or displays that were recently rotated. – EncodedNybble Mar 30 '16 at 18:03
  • I wouldn't risk recreating the VirtualDisplay **while** the device is rotating :) Be careful not to call updateTexImage on any released texture, this happened to me (as an effect of onFrameAvailable still being called) and all the subsequent frames were flickering for some reason. I kept the GL texture to be safe against any "external oes" rebindings (and also so I don't mess around with threads, I just recreate the surfacetexture on the main/whatever thread), maybe that has something to do with it? – Adrian Crețu Mar 31 '16 at 03:49