Android OpenGL Render to a frameBuffer in the original (small) resolution and then scale it (viewPort) to the screen size

Question

I have a NV21 (YUV420) camera video which I'm applying on it a fragment shader in order to get some filter effects and YUV to RGB convertion as well. Everything is working except the bad performance. My fragment shader is a bit heavy because it has many textur2D() calls.

The original frame resolution is 480x640 pixels, and I noticed that if I'm setting the viewport to this original size (instead of fullscreen size) it is working good and fluently.

So basically I need first to render the frame and processing it in a frameBuffer (FBO) with that original size and then (after the shader's work done) scale it to the fullscreen size using viewport (1080x1920 mostly), And it means that the "heavy" processing work would be applied on much less fragments.

I've found out some tutorials and similar questions here how to achieve that, but unfortunately I've got no luck with that. (Got some black screens or GL_INVALID_OPERATION and etc')... Any help would be much appreciated.

Also, another (optional) performance tweak that I don't know how to deal with (if it's possible) is to combined somehow these 3 textures (Y_tex, U_tex and V_tex) to a single texture which be uniformed to the shader as a single sampler and then I can make just one texture2D() call in the shader in order to get the current YUV values and convert them to RGB values.

This is my renderer code:

static class MyRenderer implements GLSurfaceView.Renderer
{
    int mTextureIds[] = new int[3];
    float[] mScaleMatrix = new float[16];

    private FloatBuffer mVertexBuffer;
    private FloatBuffer mTextureBuffer;
    private ShortBuffer mDrawListBuffer;

    boolean mVideoFitEnabled = true;
    boolean mVideoDisabled = false;

    // number of coordinates per vertex in this array
    static final int COORDS_PER_VERTEX = 3;
    static final int TEXTURECOORDS_PER_VERTEX = 2;

    static float mXYZCoords[] = {
            -1.0f, 1.0f, 0.0f, // top left
            -1.0f, -1.0f, 0.0f, // bottom left
            1.0f, -1.0f, 0.0f, // bottom right
            1.0f, 1.0f, 0.0f // top right
    };

    static float mUVCoords[] = {
            0, 0, // top left
            0, 1, // bottom left
            1, 1, // bottom right
            1, 0  // top right
    };

    private short mVertexIndex[] = {0, 1, 2, 0, 2, 3}; // order to draw vertices

    private final String vertexShaderCode =

              "uniform mat4 uMVPMatrix;"
            + "attribute vec4 aPosition;\n"
            + "attribute vec2 aTextureCoord;\n"
            + "varying vec2 vTextureCoord;\n"

            + "void main() {\n"
                + "  gl_Position = uMVPMatrix * aPosition;\n"
                + "  vTextureCoord = aTextureCoord;\n"
            + "}\n";


    private final String fragmentShaderCode =

              "precision mediump float;\n"
            + "uniform sampler2D Ytex;\n"
            + "uniform sampler2D Utex,Vtex;\n"
            + "varying vec2 vTextureCoord;\n"

            + "void main(void) {\n"
                + "  float nx,ny,r,g,b,y,u,v;\n"
                + "  mediump vec4 txl,ux,vx;"
                + "  nx=vTextureCoord[0];\n"
                + "  ny=vTextureCoord[1];\n"

                + "  y=texture2D(Ytex,vec2(nx,ny)).r;\n"
                + "  u=texture2D(Utex,vec2(nx,ny)).r;\n"
                + "  v=texture2D(Vtex,vec2(nx,ny)).r;\n"

                + "  y=1.1643*(y-0.0625);\n" 
                + "  u=u-0.5;\n"
                + "  v=v-0.5;\n"

                + "  r=y+1.5958*v;\n"
                + "  g=y-0.39173*u-0.81290*v;\n"
                + "  b=y+2.017*u;\n"

                // --> Bilateral blur filter code HERE <--

                + "  gl_FragColor=vec4(r,g,b,1.0);\n"
            + "}\n";



    ReentrantLock mFrameLock = new ReentrantLock();
    Frame mCurrentFrame;

    private int mProgram;
    private int mTextureWidth;
    private int mTextureHeight;
    private int mViewportWidth;
    private int mViewportHeight;

    public MyRenderer()
    {
        ByteBuffer bb = ByteBuffer.allocateDirect(mXYZCoords.length * 4);
        bb.order(ByteOrder.nativeOrder());
        mVertexBuffer = bb.asFloatBuffer();
        mVertexBuffer.put(mXYZCoords);
        mVertexBuffer.position(0);

        ByteBuffer tb = ByteBuffer.allocateDirect(mUVCoords.length * 4);
        tb.order(ByteOrder.nativeOrder());
        mTextureBuffer = tb.asFloatBuffer();
        mTextureBuffer.put(mUVCoords);
        mTextureBuffer.position(0);

        ByteBuffer dlb = ByteBuffer.allocateDirect(mVertexIndex.length * 2);
        dlb.order(ByteOrder.nativeOrder());
        mDrawListBuffer = dlb.asShortBuffer();
        mDrawListBuffer.put(mVertexIndex);
        mDrawListBuffer.position(0);
    }

    @Override public void onSurfaceCreated(GL10 gl, EGLConfig config)
    {
        GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f);

        int vertexShader = loadShader(GLES20.GL_VERTEX_SHADER, vertexShaderCode);
        int fragmentShader = loadShader(GLES20.GL_FRAGMENT_SHADER, fragmentShaderCode);

        mProgram = GLES20.glCreateProgram(); // create empty OpenGL ES Program
        GLES20.glAttachShader(mProgram, vertexShader); // add the vertex shader to program
        GLES20.glAttachShader(mProgram, fragmentShader); // add the fragment shader to program
        GLES20.glLinkProgram(mProgram);

        int positionHandle = GLES20.glGetAttribLocation(mProgram, "aPosition");
        int textureHandle = GLES20.glGetAttribLocation(mProgram, "aTextureCoord");

        GLES20.glVertexAttribPointer(positionHandle, COORDS_PER_VERTEX, GLES20.GL_FLOAT, false, COORDS_PER_VERTEX * 4, mVertexBuffer);
        GLES20.glEnableVertexAttribArray(positionHandle);
        GLES20.glVertexAttribPointer(textureHandle, TEXTURECOORDS_PER_VERTEX, GLES20.GL_FLOAT, false, TEXTURECOORDS_PER_VERTEX * 4, mTextureBuffer);
        GLES20.glEnableVertexAttribArray(textureHandle);
        GLES20.glUseProgram(mProgram);

        int i = GLES20.glGetUniformLocation(mProgram, "Ytex");                                     //            GLES20.glUniform3i(i, 0, 1, 2);
        GLES20.glUniform1i(i, 0); /* Bind Ytex to texture unit 0 */

        i = GLES20.glGetUniformLocation(mProgram, "Utex");
        GLES20.glUniform1i(i, 1); /* Bind Utex to texture unit 1 */

        i = GLES20.glGetUniformLocation(mProgram, "Vtex");
        GLES20.glUniform1i(i, 2); /* Bind Vtex to texture unit 2 */

        mTextureWidth = 0;
        mTextureHeight = 0;
    }

    static void initializeTexture(int name, int id, int width, int height)
    {
        GLES20.glActiveTexture(name);
        GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, id);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, width, height, 0, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, null);
    }

    void setupTextures(Frame frame)
    {
        if (mTextureIds[0] != 0)
        {
            GLES20.glDeleteTextures(3, mTextureIds, 0);
        }

        GLES20.glGenTextures(3, mTextureIds, 0);

        int w = frame.getWidth();
        int h = frame.getHeight();
        int hw = (w + 1) >> 1;
        int hh = (h + 1) >> 1;

        initializeTexture(GLES20.GL_TEXTURE0, mTextureIds[0], w, h);
        initializeTexture(GLES20.GL_TEXTURE1, mTextureIds[1], hw, hh);
        initializeTexture(GLES20.GL_TEXTURE2, mTextureIds[2], hw, hh);

        mTextureWidth = frame.getWidth();
        mTextureHeight = frame.getHeight();
    }

    void updateTextures(Frame frame)
    {
        int width = frame.getWidth();
        int height = frame.getHeight();
        int half_width = (width + 1) >> 1;
        int half_height = (height + 1) >> 1;
        int y_size = width * height;
        int uv_size = half_width * half_height;

        ByteBuffer bb = frame.getBuffer();
        bb.clear();  // If we are reusing this frame, make sure we reset position and limit

        if (bb.remaining() == y_size + uv_size * 2)
        {
            bb.position(0);

            GLES20.glPixelStorei(GLES20.GL_UNPACK_ALIGNMENT, 1);
            GLES20.glPixelStorei(GLES20.GL_PACK_ALIGNMENT, 1);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[0]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, width, height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);

            bb.position(y_size);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[1]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, half_width, half_height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);

            bb.position(y_size + uv_size);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE2);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[2]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, half_width, half_height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);



            int i = GLES20.glGetUniformLocation(mProgram, "width");
            GLES20.glUniform1f(i, (float) mTextureWidth);

            i = GLES20.glGetUniformLocation(mProgram, "height");
            GLES20.glUniform1f(i, (float) mTextureHeight);
        }

        else
        {
            mTextureWidth = 0;
            mTextureHeight = 0;
        }
    }

    @Override public void onSurfaceChanged(GL10 gl, int width, int height)
    {
        GLES20.glViewport(0, 0, width, height);
        mViewportWidth = width;
        mViewportHeight = height;
    }

    @Override public void onDrawFrame(GL10 gl)
    {
        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

        mFrameLock.lock();

        if (mCurrentFrame != null && !mVideoDisabled)
        {
            GLES20.glUseProgram(mProgram);

            if (mTextureWidth != mCurrentFrame.getWidth()    ||    mTextureHeight != mCurrentFrame.getHeight())
            {
                setupTextures(mCurrentFrame);
            }

            updateTextures(mCurrentFrame);

            Matrix.setIdentityM(mScaleMatrix, 0);
            float scaleX = 1.0f, scaleY = 1.0f;
            float ratio = (float) mCurrentFrame.getWidth() / mCurrentFrame.getHeight();
            float vratio = (float) mViewportWidth / mViewportHeight;

            if (mVideoFitEnabled)
            {
                if (ratio > vratio)
                {
                    scaleY = vratio / ratio;
                }

                else
                {
                    scaleX = ratio / vratio;
                }
            }

            else
            {
                if (ratio < vratio)
                {
                    scaleY = vratio / ratio;
                }

                else
                {
                    scaleX = ratio / vratio;
                }
            }

            Matrix.scaleM(mScaleMatrix, 0, scaleX * (mCurrentFrame.isMirroredX() ? -1.0f : 1.0f), scaleY, 1);

            int mMVPMatrixHandle = GLES20.glGetUniformLocation(mProgram, "uMVPMatrix");
            GLES20.glUniformMatrix4fv(mMVPMatrixHandle, 1, false, mScaleMatrix, 0);

            GLES20.glDrawElements(GLES20.GL_TRIANGLES, mVertexIndex.length, GLES20.GL_UNSIGNED_SHORT, mDrawListBuffer);
        }

        mFrameLock.unlock();
    }

    public void displayFrame(Frame frame)
    {
        mFrameLock.lock();
        if (this.mCurrentFrame != null)
        {
            this.mCurrentFrame.recycle();
        }

        this.mCurrentFrame = frame;
        mFrameLock.unlock();
    }

    public static int loadShader(int type, String shaderCode)
    {
        int shader = GLES20.glCreateShader(type);

        GLES20.glShaderSource(shader, shaderCode);
        GLES20.glCompileShader(shader);

        return shader;
    }

    public void disableVideo(boolean b)
    {
        mFrameLock.lock();

        mVideoDisabled = b;

        if (mVideoDisabled)
        {
            if (this.mCurrentFrame != null)
            {
                this.mCurrentFrame.recycle();
            }

            this.mCurrentFrame = null;
        }

        mFrameLock.unlock();
    }

    public void enableVideoFit(boolean enableVideoFit)
    {
        mVideoFitEnabled = enableVideoFit;
    }
}

score 5 · Accepted Answer · answered May 19 '16 at 16:45

Eventually I've figured it out thanks to a very talent guy who helped me with it. Here is my renderer class which has now a frameBuffer with 2 rendering passes:

static class MyRenderer implements GLSurfaceView.Renderer
{
    int mTextureIds[] = new int[4];
    float[] mScaleMatrix = new float[16];
    float[] mFilterScaleMatrix = new float[16];

    private FloatBuffer mVertexBuffer;
    private FloatBuffer mTextureBuffer;
    private ShortBuffer mDrawListBuffer;

    private IntBuffer frameBuffer;

    boolean mVideoFitEnabled = true;
    boolean mVideoDisabled = false;

    // number of coordinates per vertex in this array
    static final int COORDS_PER_VERTEX = 3;
    static final int TEXTURECOORDS_PER_VERTEX = 2;

    static float mXYZCoords[] = {
            -1.0f, 1.0f, 0.0f, // top left
            -1.0f, -1.0f, 0.0f, // bottom left
            1.0f, -1.0f, 0.0f, // bottom right
            1.0f, 1.0f, 0.0f // top right
    };

    static float mUVCoords[] = {
            0, 0, // top left
            0, 1, // bottom left
            1, 1, // bottom right
            1, 0  // top right
    };

    private short mVertexIndex[] = {0, 1, 2, 0, 2, 3}; // order to draw vertices

    private final String vertexShaderCode =

              "uniform mat4 uMVPMatrix;"
            + "attribute vec4 aPosition;\n"
            + "attribute vec2 aTextureCoord;\n"
            + "varying vec2 vTextureCoord;\n"

            + "void main() {\n"
                + "  gl_Position = uMVPMatrix * aPosition;\n"
                + "  vTextureCoord = aTextureCoord;\n"
            + "}\n";


    private final String fragmentShaderCode =

              "YUV to RGB Conversion shader HERE";



    private final String frameBufferShader =

                "MY filter effect shader HERE";



    ReentrantLock mFrameLock = new ReentrantLock();
    Frame mCurrentFrame;

    private int mProgram;
    private int mProgramFilter;
    private int mTextureWidth;
    private int mTextureHeight;
    private int mViewportWidth;
    private int mViewportHeight;

    public MyRenderer()
    {
        ByteBuffer bb = ByteBuffer.allocateDirect(mXYZCoords.length * 4);
        bb.order(ByteOrder.nativeOrder());
        mVertexBuffer = bb.asFloatBuffer();
        mVertexBuffer.put(mXYZCoords);
        mVertexBuffer.position(0);

        ByteBuffer tb = ByteBuffer.allocateDirect(mUVCoords.length * 4);
        tb.order(ByteOrder.nativeOrder());
        mTextureBuffer = tb.asFloatBuffer();
        mTextureBuffer.put(mUVCoords);
        mTextureBuffer.position(0);

        ByteBuffer dlb = ByteBuffer.allocateDirect(mVertexIndex.length * 2);
        dlb.order(ByteOrder.nativeOrder());
        mDrawListBuffer = dlb.asShortBuffer();
        mDrawListBuffer.put(mVertexIndex);
        mDrawListBuffer.position(0);

        frameBuffer = IntBuffer.allocate(1);
    }

    @Override public void onSurfaceCreated(GL10 gl, EGLConfig config)
    {
        GLES20.glDisable(GLES20.GL_DEPTH_TEST);

        GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f);

        int vertexShader = loadShader(GLES20.GL_VERTEX_SHADER, vertexShaderCode);
        int fragmentShader = loadShader(GLES20.GL_FRAGMENT_SHADER, fragmentShaderCode);
        int filterVertexShader = loadShader(GLES20.GL_VERTEX_SHADER, vertexShaderCode);
        int filterFragmentShader = loadShader(GLES20.GL_FRAGMENT_SHADER, frameBufferShader);


        mProgram = GLES20.glCreateProgram(); // create empty OpenGL ES Program
        GLES20.glAttachShader(mProgram, vertexShader); // add the vertex shader to program
        GLES20.glAttachShader(mProgram, fragmentShader); // add the fragment shader to program
        GLES20.glLinkProgram(mProgram);

        mProgramFilter = GLES20.glCreateProgram(); // create empty OpenGL ES Program
        GLES20.glAttachShader(mProgramFilter, filterVertexShader); // add the vertex shader to program
        GLES20.glAttachShader(mProgramFilter, filterFragmentShader); // add the fragment shader to program
        GLES20.glLinkProgram(mProgramFilter);

        int positionHandle = GLES20.glGetAttribLocation(mProgram, "aPosition");
        int textureHandle = GLES20.glGetAttribLocation(mProgram, "aTextureCoord");

        GLES20.glVertexAttribPointer(positionHandle, COORDS_PER_VERTEX, GLES20.GL_FLOAT, false, COORDS_PER_VERTEX * 4, mVertexBuffer);
        GLES20.glEnableVertexAttribArray(positionHandle);
        GLES20.glVertexAttribPointer(textureHandle, TEXTURECOORDS_PER_VERTEX, GLES20.GL_FLOAT, false, TEXTURECOORDS_PER_VERTEX * 4, mTextureBuffer);
        GLES20.glEnableVertexAttribArray(textureHandle);

        GLES20.glUseProgram(mProgram);

        int i = GLES20.glGetUniformLocation(mProgram, "Ytex");
        GLES20.glUniform1i(i, 3); /* Bind Ytex to texture unit 0 */

        i = GLES20.glGetUniformLocation(mProgram, "Utex");
        GLES20.glUniform1i(i, 1); /* Bind Utex to texture unit 1 */

        i = GLES20.glGetUniformLocation(mProgram, "Vtex");
        GLES20.glUniform1i(i, 2); /* Bind Vtex to texture unit 2 */


        GLES20.glUseProgram(mProgramFilter);
        i = GLES20.glGetUniformLocation(mProgramFilter, "Ytex");
        GLES20.glUniform1i(i, 0);

        mTextureWidth = 0;
        mTextureHeight = 0;
    }

    static void initializeTexture(int name, int id, int width, int height)
    {
        GLES20.glActiveTexture(name);
        GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, id);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, width, height, 0, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, null);
    }

    void setupTextures(Frame frame)
    {
        if (mTextureIds[0] != 0)
        {
            GLES20.glDeleteTextures(4, mTextureIds, 0);
        }

        GLES20.glGenTextures(4, mTextureIds, 0);

        int w = frame.getWidth();
        int h = frame.getHeight();
        int hw = (w + 1) >> 1;
        int hh = (h + 1) >> 1;

        initializeTexture(GLES20.GL_TEXTURE0, mTextureIds[0], w, h);
        initializeTexture(GLES20.GL_TEXTURE1, mTextureIds[1], hw, hh);
        initializeTexture(GLES20.GL_TEXTURE2, mTextureIds[2], hw, hh);

        GLES20.glGenFramebuffers(1, frameBuffer);
        GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, frameBuffer.get(0));

        GLES20.glActiveTexture(GLES20.GL_TEXTURE3);
        GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[3]);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
        GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, w, h, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);
        GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0, GLES20.GL_TEXTURE_2D, mTextureIds[3], 0);

        GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);

        mTextureWidth = frame.getWidth();
        mTextureHeight = frame.getHeight();



        GLES20.glUseProgram(mProgramFilter);

        int i = GLES20.glGetUniformLocation(mProgramFilter, "width");
        GLES20.glUniform1f(i, (float) mTextureWidth);

        i = GLES20.glGetUniformLocation(mProgramFilter, "height");
        GLES20.glUniform1f(i, (float) mTextureHeight);
    }

    void updateTextures(Frame frame)
    {
        int width = frame.getWidth();
        int height = frame.getHeight();
        int half_width = (width + 1) >> 1;
        int half_height = (height + 1) >> 1;
        int y_size = width * height;
        int uv_size = half_width * half_height;

        ByteBuffer bb = frame.getBuffer();
        bb.clear();  // If we are reusing this frame, make sure we reset position and limit

        if (bb.remaining() == y_size + uv_size * 2)
        {
            bb.position(0);

            GLES20.glPixelStorei(GLES20.GL_UNPACK_ALIGNMENT, 1);
            GLES20.glPixelStorei(GLES20.GL_PACK_ALIGNMENT, 1);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[0]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, width, height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);

            bb.position(y_size);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[1]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, half_width, half_height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);

            bb.position(y_size + uv_size);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE2);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureIds[2]);
            GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, half_width, half_height, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, bb);
        }

        else
        {
            mTextureWidth = 0;
            mTextureHeight = 0;
        }
    }

    @Override public void onSurfaceChanged(GL10 gl, int width, int height)
    {
        /// GLES20.glViewport(0, 0, width, height);
        mViewportWidth = width;
        mViewportHeight = height;
    }

    @Override public void onDrawFrame(GL10 gl)
    {
        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

        mFrameLock.lock();

        if (mCurrentFrame != null && !mVideoDisabled)
        {
            if (mTextureWidth != mCurrentFrame.getWidth()    ||    mTextureHeight != mCurrentFrame.getHeight())
            {
                setupTextures(mCurrentFrame);
            }

            updateTextures(mCurrentFrame);


            /// Step 1: Smoothing Filter - Render to FrameBuffer [pass 1]
            Matrix.setIdentityM(mFilterScaleMatrix, 0);
            GLES20.glViewport(0, 0, mTextureWidth, mTextureHeight);

            GLES20.glUseProgram(mProgramFilter);

            int mMVPFilterMatrixHandle = GLES20.glGetUniformLocation(mProgramFilter, "uMVPMatrix");
            GLES20.glUniformMatrix4fv(mMVPFilterMatrixHandle, 1, false, mFilterScaleMatrix, 0);

            GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, frameBuffer.get(0));

            GLES20.glDrawElements(GLES20.GL_TRIANGLES, mVertexIndex.length, GLES20.GL_UNSIGNED_SHORT, mDrawListBuffer);



            /// Step 2: Draw + RGB Conversion - Render to screen [pass 2]
            Matrix.setIdentityM(mScaleMatrix, 0);

            float scaleX = 1.0f, scaleY = 1.0f;
            float ratio = (float) mCurrentFrame.getWidth() / mCurrentFrame.getHeight();
            float vratio = (float) mViewportWidth / mViewportHeight;

            if (mVideoFitEnabled)
            {
                if (ratio > vratio)
                {
                    scaleY = vratio / ratio;
                }

                else
                {
                    scaleX = ratio / vratio;
                }
            }

            else
            {
                if (ratio < vratio)
                {
                    scaleY = vratio / ratio;
                }

                else
                {
                    scaleX = ratio / vratio;
                }
            }

            Matrix.scaleM(mScaleMatrix, 0, scaleX * (mCurrentFrame.isMirroredX() ? -1.0f : 1.0f), scaleY, 1);

            GLES20.glUseProgram(mProgram);

            GLES20.glViewport(0, 0, mViewportWidth, mViewportHeight);

            GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);

            int mMVPMatrixHandle = GLES20.glGetUniformLocation(mProgram, "uMVPMatrix");
            GLES20.glUniformMatrix4fv(mMVPMatrixHandle, 1, false, mScaleMatrix, 0);

            GLES20.glDrawElements(GLES20.GL_TRIANGLES, mVertexIndex.length, GLES20.GL_UNSIGNED_SHORT, mDrawListBuffer);
        }

        mFrameLock.unlock();
    }

    public void displayFrame(Frame frame)
    {
        mFrameLock.lock();
        if (this.mCurrentFrame != null)
        {
            this.mCurrentFrame.recycle();
        }

        this.mCurrentFrame = frame;
        mFrameLock.unlock();
    }

    public static int loadShader(int type, String shaderCode)
    {
        int shader = GLES20.glCreateShader(type);

        GLES20.glShaderSource(shader, shaderCode);
        GLES20.glCompileShader(shader);

        return shader;
    }

    public void disableVideo(boolean b)
    {
        mFrameLock.lock();

        mVideoDisabled = b;

        if (mVideoDisabled)
        {
            if (this.mCurrentFrame != null)
            {
                this.mCurrentFrame.recycle();
            }

            this.mCurrentFrame = null;
        }

        mFrameLock.unlock();
    }

    public void enableVideoFit(boolean enableVideoFit)
    {
        mVideoFitEnabled = enableVideoFit;
    }
}

Michael IV · Answer 2 · 2016-05-14T08:59:40.640

What you are trying to do is called downsampling.What you need to do is to render first into a small FBO, where both the viewport and the FBO attachment are of the same size. In that pass you can apply your blur effect. Then you can render or blit it to another FBO of the original size to get the texture scaled back.It's important to note that depending on your blue technique, the upscaled result may have noticeable degraded quality.

As some OpenGL API calls may have different namings in Android SDK,here is the general pseudo-code for what you need to do:

Let's w = original width and h = original height.

1.Create a custom FBO with texture attachment of the size w/2 and h/2 (if you plan to downsample at half the original resolution.

2.Attach the texture to the FBO.Bind the FBO for write.

3.set glViewport() to the same size as the FBO texture attachment.

4.Render a full screen quad.Apply your effect during this pass.

5.Second pass:Bind back the default framebuffer(alternatively - another custom FBO if you have other rendering stage later).If you just want to blit,then make sure you bind the fist FBO for read and not for write.

6.Bind the texture attached to the FBO from the previous pass to the sampler unit.

7.Set the viewport to w and h size.Draw the full screen quad.Do in the fragment shader whatever you need.(Gamma corrections,blending etc..)

That's it.What is nice in this technique is that OpenGL does the upscale/downscale for you automatically with the filtering type you select for the texture being involved in the process.

And on the side note:If you have performance issues this is not a good idea to write a renderer in Java.Though it is possible your issues are due to wrong API usage or inefficient algorithm application for the blur effect.

Thank you. As I mentioned above I know what should be done, I've tried using FBO to do that but didn't succeed, That's why I asked for a bit more specific answer regarding to my renderer — Eliran Kuta, May 13 '16 at 19:25
There is nothing in your code even close to what I have explained in my answer.Do you know what is offscreen rendering?Do you know how to render into custom FBO?I am not going to reiterate on this here as there are plenty online resources on this subject. — Michael IV, May 14 '16 at 08:45
I know I don't have FBO in my code , and I know what is off screen rendering, and as I said, I know what should be done to solve it, I just didn't succeed doing it practically. The code above is working without using frame buffers. That's exactly what I've asked, How to integrate FBOs to my renderer — Eliran Kuta, May 14 '16 at 09:06
But you are keeping telling me you know what to do.Post the whole code including FBO setup part if you want someone to help you. — Michael IV, May 14 '16 at 12:11

Android OpenGL Render to a frameBuffer in the original (small) resolution and then scale it (viewPort) to the screen size

2 Answers2

Linked