How to improve opengl es display performance in android

Question

I am working to improve the video display efficiency of our video conferencing project in android. Video display is implemented in native opengl code. The opengl code is implemented in native with opengl version 1. The code given below is used for displaying each frames of the video.

int ofi_vc_video_display::render_video_frame(unsigned char *frame_buffer)
{

    // Check the frame is available or not. If available display the frame.
    if (frame_buffer != NULL){

        // Clear the screen buffers.
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        // Bind the frame data to the texture.
        glTexSubImage2D(GL_TEXTURE_2D,
                        0,
                        0,
                        0,
                        frame_width,
                        frame_height,
                        GL_RGB,
                        GL_UNSIGNED_BYTE,
                        frame_buffer);

        // Check for the error status.
        while ((gl_error_status=glGetError()) != GL_NO_ERROR) {

            error_status = gl_error_status;
          }

        // Transform and rotate the texture.
        glMatrixMode(GL_TEXTURE);
        glLoadIdentity();
        glTranslatef(0.5, 0.5, 0.0);
        glRotatef(180.0, 180.0, 0.0,1.0);
        glTranslatef(-0.5, -0.5, 0.0);
        glMatrixMode(GL_MODELVIEW);

        glLoadIdentity();
        glTranslatef(0, 0, -4);

        // Enable drawing texture.
        glBindTexture(GL_TEXTURE_2D, mTextureId);

        // Draw the frames in texture.
        glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_SHORT, indices);

        // Check for the error status.
        while ((gl_error_status=glGetError()) != GL_NO_ERROR) {

            error_status = gl_error_status;
          }
    }
    return error_status;
}

All initializations are done before. The code is fine for lower resolutions. But when displaying higher resolutions like 640 X 480, glTexSubImage2D itself is taking around 35 - 40 ms and the whole display time goes above 50 ms per frame. But I need 30 fps.

Can some one help with your suggestions.

Which device and version of Android? How large are your video frames? There's a texture upload speed test in Grafika (https://github.com/google/grafika); what value do you get from that? — fadden, Apr 25 '14 at 14:35
@fadden The above data is from amazone kindle fire hdx with android 4.2.2. And i cant run your graphika since it require 4.3. I tested in Motorola moto G with 4.4.2, there glTexSubImage2D is taking more than 50 - 65ms per frame and the texture upload speed test is giving 3807 us per iteration... — Kevin K, Apr 28 '14 at 05:55
@fadden Is there any other way to speed up the frame display.... — Kevin K, Apr 28 '14 at 07:12
The upload speed test is using a 512x512 RGBA texture. The number you got from it is reasonable (I get 1600us on N5/N10). Are you creating a power-of-two texture and loading the frame data into it? (I'm wondering if glTexImage2D offers a speed advantage over glTexSubImage2D... but you'd have to copy the frame to a POT buffer first.) — fadden, Apr 28 '14 at 17:11
Is there another faster way of video frame display in android other than opengl ???? — Kevin K, Apr 29 '14 at 07:30
@fadden I have tried your suggestion by copying the frame to a power of two buffer and then calling glTexImage2D but to find no improvements in kindle fire hdx(adreno 330 GPU) and in moto g(adreno 305 GPU). But improvement is there in sony experia C(4.2.2)(SGX544 GPU).... — Kevin K, Apr 29 '14 at 12:55
You can handle video quickly with MediaCodec and GLES, but you need API 16+ (Android 4.1), and your video stream has to have a suitable codec. API 18 for best results. (See the various activities in Grafika, e.g. "double decode", which plays two videos side-by-side.) — fadden, Apr 29 '14 at 15:07
@fadden Thanks for your reply... I have already implemented the media codec at display side but the hardware decoders available in different devices are different and that too it requires 4.1 plus. But my app need to support from 2.3 onwards... I was looking into it so that devices which do not support mediacodec also produces 30 fps for 480p....... — Kevin K, Apr 30 '14 at 04:02

score 5 · Answer 1 · edited Apr 25 '14 at 14:37

5

glTexSubImage2D() is too slow for video frame rates. With OpenGL ES 1.1 and native code you can load the video frames into textures much faster by avoiding that and using the EGL Image Extensions instead. It is discussed with example code in this article.

With OpenGL ES 2.0, you could also perform the YUV to RGB color space conversion in shader code which is also a major performance improvement for video. I have posted examples of this here on StackOverflow before.

edited Apr 25 '14 at 14:37

fadden

51,356
5
116
166

answered Apr 17 '14 at 15:45

ClayMontgomery

2,786
1
15
14

@ ClayMontgomery.. Thanks for your replay and the detailed article.. I am trying to implement it. But the article says GraphicsBuffer is available only with framework apis not in ndk. But I am writing it in native code. What can I do to build it with ndk. Sorry if the doubt is silly but I'm not clear on this.. – Kevin K Apr 23 '14 at 10:03
That statement is wrong and I think from a different blog. Listing 5 in my article shows how to use a GraphicBuffer with native code. – ClayMontgomery Apr 23 '14 at 18:08
@ ClayMontgomery. Thanks for the reply.. But I am not able to build my code with ndk-r9d. It is saying GraphicBuffer is not declared in this scope. Is there another header file to be added ? The same comments for eglCreateImageKHR and glEGLImageTargetTexture2DOES.... – Kevin K Apr 24 '14 at 05:14
@ ClayMontgomery // Write 2D image to pBitmap For this line of your example I have used a memcpy from source frame array to bitmap array. Hope that would be fine... But still I am not able to build it....Should I do something in make file... – Kevin K Apr 24 '14 at 09:38
@ ClayMontgomery If you have a working source code of it please give me a link to it.... – Kevin K Apr 24 '14 at 12:45
@ ClayMontgomery Can I have a help – Kevin K Apr 25 '14 at 05:54
@jacob with reference to your question at http://stackoverflow.com/questions/21151259/replacing-glreadpixels-with-egl-khr-image-base-for-faster-pixel-copy/. How did you add the header file GraphicBuffer.h for building it using ndk... – Kevin K Apr 25 '14 at 06:13
1

The linked article is correct -- the GraphicBuffer class is not part of the NDK, and is not a public API. Any code that uses it may break when a new version of Android is released. The header file can be obtained by downloading the AOSP sources; visit developer.android.com. You can't just grab https://android.googlesource.com/platform/frameworks/native/+/kitkat-release/include/ui/GraphicBuffer.h though, you also need all of its dependencies. – fadden Apr 25 '14 at 14:41
That is correct. I meant that GraphicBuffer can be used in your native code, but it is not particularly easy to implement. You must copy the required parts of the AOSP sources. – ClayMontgomery Apr 25 '14 at 16:12

How to improve opengl es display performance in android

1 Answers1

Linked