0

My aim:
Use filters (cropping, Black and white, Edge detection) on a MP4 video from the SD card using render script.

Attempted Solutions
-Use MediaCodec to output to a surface directly.
The rendered colour were correct but I could not find a way to process each frame at a time to apply filters using renderscript.

-Copy the decoded buffer from render script and convert to RGB using ScriptIntrinsicYuvToRGB
I can not using ScriptIntrinsicYuvToRGB because it assumes the incoming YUV data is formatted in a different way from the incoming data. (How do I correctly convert YUV colours to RGB in Android?)

-Copy the decoded buffer from render script and convert to RGB using custom code
My current solution converts the YUV to RGB using code very similar to https://stackoverflow.com/a/12702836/601147 (Thanks @Derzu)

            /**
         * Converts YUV420 NV21 to RGB8888
         *
         * @param data byte array on YUV420 NV21 format.
         * @param width pixels width
         * @param height pixels height
         * @return a RGB8888 pixels int array. Where each int is a pixels ARGB.
         */
        public int[] convertYUV420_NV21toRGB8888(byte [] data, int width, int height) {
            int size = width*height;
            int offset = size;
            int[] pixels = new int[size];
            int u, v, y1, y2, y3, y4;

            // i percorre os Y and the final pixels
            // k percorre os pixles U e V
            for(int i=0, k=0; i < size; i+=2, k+=2) {
                y1 = data[i]&0xff;
                y2 = data[i+1]&0xff;
                y3 = data[width+i  ]&0xff;
                y4 = data[width+i+1]&0xff;

                u = data[offset+k  ]&0xff;
                v = data[offset+k+1]&0xff;
                u = u-128;
                v = v-128;

//                pixels[i] = convertYUVtoRGB(y1, v,u);
//                pixels[i+1] = convertYUVtoRGB(y2, v,u);
//                pixels[width+i  ] = convertYUVtoRGB(y3, v,u);
//                pixels[width+i+1] = convertYUVtoRGB(y4, v,u);

                pixels[i] = convertYUVtoRGB(y1, u, v);
                pixels[i+1] = convertYUVtoRGB(y2,  u, v);
                pixels[width+i  ] = convertYUVtoRGB(y3,  u, v);
                pixels[width+i+1] = convertYUVtoRGB(y4, u, v);

                if (i!=0 && (i+2)%width==0)
                    i+=width;
            }

            return pixels;
        }

        private int convertYUVtoRGB(int y, int u, int v) {
            int r,g,b;

            r = y + (int)1.402f*v;
            g = y - (int)(0.344f*u +0.714f*v);
            b = y + (int)1.772f*u;
            r = r>255? 255 : r<0 ? 0 : r;
            g = g>255? 255 : g<0 ? 0 : g;
            b = b>255? 255 : b<0 ? 0 : b;
            return 0xff000000 | (b<<16) | (g<<8) | r;
        }

The only difference I made was to flip the UV values from

pixels[i] = convertYUVtoRGB(y1, v,u);

to

pixels[i] = convertYUVtoRGB(y1, u, v);

Because the latter one works better.

My problem
The colours look ok-ish on my Huddle 2, but they are totally wrong on the Genymotion JellyBean emulator.

Huddle2colour being rendered on my Huddle 2 ok-ish

Genymotion Genymotion rendering the colours totally wrong

Please can you offer me help and suggestions?
Thanks you very much

Community
  • 1
  • 1
dewijones92
  • 1,319
  • 2
  • 24
  • 45
  • If your filters are relatively simple you can write them with GLES shaders, e.g. https://www.youtube.com/watch?v=kH9kCP2T5Gg . Otherwise your best bet is to render the Surface offscreen with GLES to do the color conversion, then read it back with `glReadPixels()`. AFAIK the renderscript conversion is only expected to work with Camera output, which is either YV12 or NV21 -- neither of which MediaCodec uses. – fadden May 10 '15 at 15:49

1 Answers1

3

You need to check which color format it actually uses - this assumes that the output is NV12, but it looks like your output is I420 or something similar (planar, not semiplanar). Have a look at http://bigflake.com/mediacodec/, in particular https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java (the checkFrame method).

The key element is this:

int colorFormat = format.getInteger(MediaFormat.KEY_COLOR_FORMAT);

The checkFrame method also shows you how to deal with it if it is planar instead of semiplanar, which should work for your case.

Do keep in mind that devices are also allowed to output in proprietary formats, that you can't interpret as simple as this - you need to be ready to handle that case in some way (e.g. telling the user that your app can't handle it, fall back to other SW based implementations, etc).

mstorsjo
  • 12,983
  • 2
  • 39
  • 62