How can I process the frames from camera using SurfaceView android

Question

I am working in project where I capture frames fps from Android camera. after each frame is converted in Bitmap to use a native library with opencv SDK and after this bitmap is showed in an ImageView. this process to consume heap memory and to affect the performance the app. I add the next tag in the manifest.xml to increase the heap memory:

<application
    android:name="app"
    android:largeHeap="true" />

With this, the app work fine when to generate an apk version in debug mode, but when to generate an apk version in release mode, the app is very slow and doesn't work fine, there is issues in the performance.

I worked with a logic using a SurfaceView component to get frames from camera, I use a SurfaceView because I need to use in background inside of the app, I want to ask you, how do I solve this problem, counting on your help.

I am using this logic to get frames:

public synchronized void onPreviewFrame(final byte[] data, final Camera camera) {
        executorService.execute(new Runnable() {
            public void run() {

      Camera.Parameters parameters = camera.getParameters();
      orientation = getOrientationDevice().getOrientation();
      orientation = normalize(orientation);

      //converter yuv to bitmap
      int width = parameters.getPreviewSize().width;
      int height = parameters.getPreviewSize().height;
      YuvImage yuv = new YuvImage(data,PreviewFormat,width, height, null);
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      yuv.compressToJpeg(new Rect(0, 0, width, height), 80, out);
      byte[] bytes = out.toByteArray();
      Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

               // rotate image 
             if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT){
                    matrix.setScale(-1, 1);
                    matrix.postTranslate(bitmap.getWidth(), 0);
                    matrix.postRotate(rotation + 90);
                } else {
                    matrix.postRotate(rotation + 90);
                }                        

               // process bitmap from method jni
                ....

               // to pass bitmap a ImageView to show
               mCameraFrame.onFrame(bitmap.copy(bitmap.getConfig(), false));


            if (bitmap != null && !bitmap.isRecycled()) {
                bitmap.recycle();
                bitmap = null;
            }


            camera.addCallbackBuffer(data);

            return;

            });

        }

    };

in the UI the callback receive the bitmap to show in Imageview

@Override
    public void onFrame(final Bitmap mBitmap) {

        this.bitmapWeakReference = new WeakReference<Bitmap>(mBitmap);

        if (imageViewReference != null && bitmapWeakReference.get() != null) {
            final ImageView imageView = imageViewReference.get();
            if (imageView != null) {

                imageView.post(new Runnable() {
                    @Override
                    public void run() {
                        imageView.setImageBitmap(bitmapWeakReference.get());
                    }
                });

            }
        }

Have you looked at the examples that come with the opencv android sdk? I believe the examples have code to handle all of this for you, and provide you with Mat to do any opencv operations you wish on each frame. — medloh, Dec 22 '15 at 22:33
hi, I was trying to use opencv to get frame, but with this sdk had issues to show the image in different device orientation, because I am trying to use the api camera Android. — fahoyos127, Dec 23 '15 at 14:24

score 1 · Accepted Answer · edited May 23 '17 at 11:54

1

Working with camera API at high frame rate requires to push camera callbacks off the main UI thread, you will need a separate Handler thread for that. Get rid of all unnecessary processing in your onPreviewFrame() callback. You can pre-calculate the camera parameters. Avoid new, even for Rect: garbage collection may be killing performance.

You can feed a YUV frame via JNI to native OpenCV-based JNI and all transformations and conversions may be performed much faster.

If you don't use OpenCV for every bitmap, you can convert YUV to RGB without going through JPEG. Rotation by 90° can be performed in one step with such conversion.

If you are running on a multi-core device, you can start the executorService with newFixedThreadPool(), but then release of camera buffers (calls to camera.addCallbackBuffer()) may require delicate management, especially if you want the frame rate to be uniform.

If you simply want to show two copies of the preview stream, you can use OpenGL, and CPU will not be involved in rotation and YUV to RGB conversion at all. You can even apply some advanced image processing to texture displayed via OpenGL.

edited May 23 '17 at 11:54

Community

1
1

answered Dec 24 '15 at 09:55

Alex Cohn

56,089
9
113
307

Thanks for suggestions, can you help about rotate and transpose Mat operations, where can I find examples? please. – fahoyos127 Dec 26 '15 at 04:54
What is your problem with Mat operations? – Alex Cohn Dec 26 '15 at 06:52
How can I do the rotations operation and translate left side to right side of image by mirror effect in the image?. is it more efficient to do these operations in Android Java or JNI Native? – fahoyos127 Dec 28 '15 at 13:38
You can use [the answer on SO](http://stackoverflow.com/a/16278334/192373). OpenCV performs these operations much better than whatever one can achieve in Java, but if you don't need RGB and OpenCV, you won't gain from these extra layers of complexity. – Alex Cohn Dec 28 '15 at 14:03

How can I process the frames from camera using SurfaceView android

1 Answers1