Camera preview image data processing with Android L and Camera2 API

Question

I'm working on an android app that is processing the input image from the camera and displays it to the user. This is fairly simple, I register a PreviewCallback on the camera object with the setPreviewCallbackWithBuffer. This is easy and works smoothly with the old camera API

public void onPreviewFrame(byte[] data, Camera cam) {
    // custom image data processing
}

I'm trying to port my app to take advantage of the new Camera2 API and I'm not sure how exactly shall I do that. I followed the Camera2Video in L Preview samples that allows to record a video. However, there is no direct image data transfer in the sample, so I don't understand where exactly shall I get the image pixel data and how to process it.

Could anybody help me or suggest the way how one can get the the functionality of PreviewCallback in android L, or how it's possible to process preview data from the camera before displaying it to the screen? (there is no preview callback on the camera object)

Thank you!

Yes, I did. Check VP's response and also Camera2Basic and Camera2Video from android samples. You need to create an ImageReader and use `setOnImageAvailableListener` to get a new image when captured. In order to draw image I created OpenGL surface which renders texture and a shader that converts `YUV_420_888` to `RGB`. — bubo, Dec 20 '15 at 15:52
Thanks, I have checked both repos and VP response. When I set addTarget(mImageReader.getSurface()); It gives only three frames onImageAvailable after that it freezes the preview. — user1154390, Dec 20 '15 at 16:59
This (or something similar) can happen when you don't read/close the image from the ImageReader. Make sure that in the listener `onImageAvailable(...)` you do read and close the image. The Listener cannot be empty, even if you don't use the image you need to read it (for example with `reader.acquireNextImage()`). — bubo, Dec 25 '15 at 11:26
Thanks @bubo. It took me a while to figure out this behavior because it was not mentioned in documentation. — user1154390, Jan 04 '16 at 10:01

score 34 · Answer 1 · answered Apr 22 '17 at 21:07

Combining a few answers into a more digestible one because @VP's answer, while technically clear, is difficult to understand if it's your first time moving from Camera to Camera2:

Using https://github.com/googlesamples/android-Camera2Basic as a starting point, modify the following:

In createCameraPreviewSession() init a new Surface from mImageReader

Surface mImageSurface = mImageReader.getSurface();

Add that new surface as a output target of your CaptureRequest.Builder variable. Using the Camera2Basic sample, the variable will be mPreviewRequestBuilder

mPreviewRequestBuilder.addTarget(mImageSurface);

Here's the snippet with the new lines (see my @AngeloS comments):

private void createCameraPreviewSession() {

    try {

        SurfaceTexture texture = mTextureView.getSurfaceTexture();
        assert texture != null;

        // We configure the size of default buffer to be the size of camera preview we want.
        texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());

        // This is the output Surface we need to start preview.
        Surface surface = new Surface(texture);

        //@AngeloS - Our new output surface for preview frame data
        Surface mImageSurface = mImageReader.getSurface();

        // We set up a CaptureRequest.Builder with the output Surface.
        mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);

        //@AngeloS - Add the new target to our CaptureRequest.Builder
        mPreviewRequestBuilder.addTarget(mImageSurface);

        mPreviewRequestBuilder.addTarget(surface);

        ...

Next, in setUpCameraOutputs(), change the format from ImageFormat.JPEG to ImageFormat.YUV_420_888 when you init your ImageReader. (PS, I also recommend dropping your preview size for smoother operation - one nice feature of Camera2)

mImageReader = ImageReader.newInstance(largest.getWidth() / 16, largest.getHeight() / 16, ImageFormat.YUV_420_888, 2);

Finally, in your onImageAvailable() method of ImageReader.OnImageAvailableListener, be sure to use @Kamala's suggestion because the preview will stop after a few frames if you don't close it

    @Override
    public void onImageAvailable(ImageReader reader) {

        Log.d(TAG, "I'm an image frame!");

        Image image =  reader.acquireNextImage();

        ...

        if (image != null)
            image.close();
    }

I got `java.lang.IllegalArgumentException: submitRequestList:216: Request targets Surface that is not part of current capture session`, but using Google's [CameraView](https://github.com/google/cameraview) instead of Camera2Basic. — Sira Lam, May 02 '18 at 05:07
In the above code snippet, one can process/modify the image inside the `onImageAvailable` function. This, however, won't display the image in preview. Right? How to display the image on the preview(`TextureView`) after processing is done? — vyi, Aug 19 '18 at 12:13
I have tried your approach, But preview gets paused and receiving only a few callbacks of onImageAvailable. — Bipin Vayalu, Jan 08 '19 at 11:11
The defined surface needs to be added to the list send to createCaptureSession - just like the preview surface already there - that is if you have a preview showing. — slott, Jun 09 '19 at 15:56

score 30 · Accepted Answer · edited Dec 17 '16 at 02:06

30

Since the Camera2 API is very different from the current Camera API, it might help to go through the documentation.

A good starting point is camera2basic example. It demonstrates how to use Camera2 API and configure ImageReader to get JPEG images and register ImageReader.OnImageAvailableListener to receive those images

To receive preview frames, you need to add your ImageReader's surface to setRepeatingRequest's CaptureRequest.Builder.

Also, you should set ImageReader's format to YUV_420_888, which will give you 30fps at 8MP (The documentation guarantees 30fps at 8MP for Nexus 5).

edited Dec 17 '16 at 02:06

answered Sep 21 '14 at 18:41

VP.

1,200
13
20

Hello, @VP. Could you please advise me how to convert YUV_420_888 Image, that I receive in onImageAvailable listener, to Bitmap? – mol Feb 09 '15 at 15:12
Here you go: Size largest = Collections.max(Arrays.asList(map.getOutputSizes(ImageFormat.YUV_420_888)), new CompareSizesByArea()); Instead of .JPEG -- this is from the camera2Basic repo base. – Sipty Mar 27 '15 at 12:32
1

@Ruban I added a response with code below based on VP's answer to clarify this implementation: http://stackoverflow.com/a/43564630/630996 – AngeloS Apr 22 '17 at 21:11
1

Even while using `YUV_420_888` the preview of the camera is significantly lagging (in comparison with Camera1's `PreviewCallback`) and producing no more than 10 FPS on Moto G3 (while producing more than 30 FPS with same resolution on Camera1). Is it a known issue? – Dmitry Zaytsev Apr 27 '17 at 20:25
If you suffer low frame rate by Camera2 check this answer https://stackoverflow.com/a/51083567/2606068 good luck.@DmitryZaytsev – wdanxna Jun 28 '18 at 13:02
**ImageReader to get JPEG** and **ImageReader's format to YUV_420_888** doesn't make sense – user25 Jul 29 '18 at 10:50
it's slow https://stackoverflow.com/questions/51579859/camera2-set-preview-view-and-get-preview-callbacks – user25 Jul 29 '18 at 12:44
@VP I have followed your approach, But still no success. I am have tried on TEMPLATE_RECORD and TEMPLATE_PREVIEW as well. – Bipin Vayalu Jan 08 '19 at 11:00
I am setting up 2 surfaces one is Preview and other ImageReader (with YUV_420_888 format). As and when I start preview. Preview paused and received 5-7 callback on the onImageAvailable. – Bipin Vayalu Jan 08 '19 at 11:08

score 16 · Answer 3 · answered Jul 22 '15 at 09:07

16

In the ImageReader.OnImageAvailableListener class, close the image after reading as shown below (this will release the buffer for next capture). You will have to handle exception on close

      Image image =  imageReader.acquireNextImage();
      ByteBuffer buffer = image.getPlanes()[0].getBuffer();
      byte[] bytes = new byte[buffer.remaining()];
      buffer.get(bytes);
      image.close();

answered Jul 22 '15 at 09:07

Kamala

161
1
2

This is a very useful snippet – twerdster May 25 '16 at 07:07
How can you do that on each frame ? – Florian Mac Langlade May 30 '16 at 15:42
This is only corrent when you use JPEG format to create ImageReader, or there will be more than one planes in the image – Charlesjean Feb 09 '18 at 08:57

score 9 · Answer 4 · answered Mar 20 '15 at 08:20

I needed the same thing, so I used their example and added a call to a new function when the camera is in preview state.

private CameraCaptureSession.CaptureCallback mCaptureCallback
            = new CameraCaptureSession.CaptureCallback()
    private void process(CaptureResult result) {
        switch (mState) {
            case STATE_PREVIEW: {
                    if (buttonPressed){
                        savePreviewShot();
                    }
                break;
            }

The savePreviewShot() is simply a recycled version of the original captureStillPicture() adapted to use the preview template.

   private void savePreviewShot(){
        try {
            final Activity activity = getActivity();
            if (null == activity || null == mCameraDevice) {
                return;
            }
            // This is the CaptureRequest.Builder that we use to take a picture.
            final CaptureRequest.Builder captureBuilder =
                    mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
            captureBuilder.addTarget(mImageReader.getSurface());

            // Orientation
            int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
            captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, ORIENTATIONS.get(rotation));

            CameraCaptureSession.CaptureCallback CaptureCallback
                    = new CameraCaptureSession.CaptureCallback() {

                @Override
                public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request,
                                               TotalCaptureResult result) {
                    SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd_HH:mm:ss:SSS");
                    Date resultdate = new Date(System.currentTimeMillis());
                    String mFileName = sdf.format(resultdate);
                    mFile = new File(getActivity().getExternalFilesDir(null), "pic "+mFileName+" preview.jpg");

                    Log.i("Saved file", ""+mFile.toString());
                    unlockFocus();
                }
            };

            mCaptureSession.stopRepeating();
            mCaptureSession.capture(captureBuilder.build(), CaptureCallback, null);
        } catch (Exception e) {
            e.printStackTrace();
        }
    };

I'm using your solution. The problem is that the camera2basic application (https://github.com/googlesamples/android-Camera2Basic), gets stuck after the first image capture. Did you solve this? — user2924714, Jul 06 '15 at 07:14
In addition, the image saving should be done outside the Ui thread — user2924714, Jul 06 '15 at 07:15
Yes, I changed the line mCaptureSession.capture(captureBuilder.build(), CaptureCallback, null); to mCaptureSession.capture(captureBuilder.build(), mCaptureCallback, mBackgroundHandler); — panonski, Jul 06 '15 at 12:37
is savePreviewShot actually saving the preview frame that is visible, or is it capturing the next available frame? createCaptureRequest implies future? — RoundSparrow hilltx, Apr 25 '16 at 15:45
A single preview frame spends very little time being visible. :) The frames are flowing all the time (approx. 30 frames/sec on my fastest device). However, the device can manage to save only few of them (eg. 5-6), so you could say that 1 in 6 frames gets saved. — panonski, Apr 26 '16 at 06:57
In your onCaptureCompleted how do you actually save the data? — twerdster, May 25 '16 at 07:11
You don't save it there. You can use `ImageReader` for that. When an image is available you can trigger a separate thread to save the file, so your main thread is not burdened with it. You can find plenty of examples online. — panonski, May 25 '16 at 07:36

score 2 · Answer 5 · answered Mar 27 '18 at 03:00

It's better to init ImageReader with max image buffer is 2 then use reader.acquireLatestImage() inside onImageAvailable().

Because acquireLatestImage() will acquire the latest Image from the ImageReader's queue, dropping older one. This function is recommended to use over acquireNextImage() for most use-cases, as it's more suited for real-time processing. Note that max image buffer should be at least 2.

And remember to close() your image after processing.

Camera preview image data processing with Android L and Camera2 API

5 Answers5

Linked