Android Camera2: Update UI between image captures with captureBurst is not synchronized

Question

I am trying to capture images with Camera2 API on Android with captureBurst() method. I start the burst capture in my button's OnClickListener:

button?.setOnClickListener(View.OnClickListener {
            imageRequestBuilder?.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE)
            imageRequest = imageRequestBuilder?.build()
            val requests = Collections.nCopies(10, imageRequest)
            cameraCaptureSession?.captureBurst(requests, captureCallback, backgroundHandler)
        })

In my OnImageAvailableListener listener, I am then reading the captured image, saving it and closing. At this point I would like to display something on my screen and I would like to it in between every captured image in the burst sequence. So I have written this code:

override fun onImageAvailable(reader: ImageReader?) {
    val image = reader?.acquireNextImage()
    if(image!=null) {
        synchronized(uiUpdate) {
            runOnUiThread(uiUpdate)
            (uiUpdate as java.lang.Object).wait()
        }

        val buffer = image?.planes?.get(0)?.buffer
        val bytes = ByteArray(buffer?.capacity()!!)
        buffer.get(bytes)
        SaveImageTask().execute(bytes)
    }

    image?.close()
}

private val uiUpdate = object : Runnable {
    override fun run() {
        synchronized(this) {
            textView?.text = "" + ++imageNum
            (this as java.lang.Object).notify()
        }
    }
}

To test it and to be sure that this is realy happening, I am using my front camera, and holding a mirror above my device so that camera actually captures the app's display. Unfortunately, the display and captures are not synchronized. I have triend to put 10 same ImageRequests to captureBurst method, but when I overview the obtained images, first three are the same (display is still displaying initial number 0), but the rest are fine (display is changing synchronosly). Why can't all be synchronized, i.e. why are first few images captured almost at the same time (at the start of the burst), and others are fine?

I have created my ImageReader with maximum of 1 Image in order to not to capture more than one image at the time:

imageReader = ImageReader.newInstance(/*some width*/, /*some height*/, /*some format*/, 1)

And the RequestBuilder is created with TEMPLATE_PREVIEW mode in order to maximize capturing screen.

I am aware that I could call capture method every time I capture new image in my OnImageAvailableListener listener, and this is working (tested!). But I need to get my application to capture images in fastest time possible.

Any help?

EDITED: This is my log as @alex-cohn has suggested:

D/onImageAvailable: display #0 Timestamp #0: 111788230978655
D/onImageAvailable: display #1 Timestamp #1: 111788264308655
D/onImageAvailable: display #2 Timestamp #2: 111788297633655
D/onImageAvailable: display #3 Timestamp #3: 111788730892655
D/onImageAvailable: display #4 Timestamp #4: 111788930856655
D/onImageAvailable: display #5 Timestamp #5: 111789030840655
D/onImageAvailable: display #6 Timestamp #6: 111789097494655
D/onImageAvailable: display #7 Timestamp #7: 111789264133655
D/onImageAvailable: display #8 Timestamp #8: 111789364112655
D/onImageAvailable: display #9 Timestamp #9: 111789464097655

EDITED2:

I have managed to get one image per one displayed number, but I am dropping 2 images every time after I had captured one, i.e.:

private val onImageAvailableListener = ImageReader.OnImageAvailableListener {reader ->
        val image = reader?.acquireLatestImage()

        if(image!=null) {
            capturedImagesCount++

            if(capturedImagesCount % frameDrop == 1) {
                Log.i("IMAGE_TIME", "Image available at: "+ SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS", Locale.getDefault()).format(Date()))
                synchronized(uiUpdate) {
                    runOnUiThread(uiUpdate)
                    (uiUpdate as java.lang.Object).wait()
                }

                val buffer = image.planes?.get(0)?.buffer
                val bytes = ByteArray(buffer?.capacity()!!)
                buffer.rewind()
                buffer.get(bytes)
                SaveImageTask().execute(bytes)
            }
        }

        image?.close()
    }

here, the frameDrop is set to 3. Also, maxImages in creating the imageReader is also set to 3. I had tested it on the one other device which is taking 2 images at the time, so in that device's case I had to set frameDrop and maxImages to 2. The downside of this is that it is still a bit slow, after all it is taking every third(second) image taken in burst mode which is not exctly the thing you want when using captureBurst method.

I still do not quite understand why this is working this way and why is camera taking images in something like pairs or triplets.

The times are about the same, but with `captureBurst` it is a bit faster (50-100ms faster I would say) — vp24, Apr 24 '19 at 16:20
Less than 100ms for 10 images? This makes sense in light of the timestamps that you posted before. The two images that were captured ”too early" had 33ms delay instead of 100ms — Alex Cohn, Apr 24 '19 at 18:24
No, I meant that taking images with `captureBurst` is taking 100ms less per one image than with multiple `capture` calls — vp24, Apr 26 '19 at 07:53
Yes. But this is still slow for me. I mean, if I know that I am capturing images atvery fast time, I wouldn't like to discart every two of three. I would like to use every but I need them to be valid (one number per one picture) — vp24, Apr 26 '19 at 08:35

Alex Cohn · Answer 1 · 2019-04-24T06:31:50.733

Setting ImageReader.maxImages to 1 does not help, because it only effects the way your app receives the images from camera (it will only receive one at a time), but this does not give you full control over the internal behaviour of the camera.

When the burst session starts, the camera (outside your app process) loads as much as possible (in you case, 3 images), while passing the images to your app (across process boundaries) one at a time. This cross-process work takes time, and only after that you have a chance to wait on the camera callback thread for UI update.

You are lucky to have caught this behaviour on your development device. On other devices, the camera internals may be different, and it may only buffer one image (as you would expect), or even more than 3. This number may also depend on the image resolution.

If you desperately need to tightly synchronize the images with UI, you need a way to drop first few images. How many? You may need to check for each device and image size.

You may find that ImageReader.acquireLatestImage() (having set maxImages to 3) will work better for you.

To have better control over the process, you could have something like

private var imageConsumedCount = 0
private var imageDisplayedCount = 0
override fun onImageAvailable(reader: ImageReader?) {
    val image = reader?.acquireNextImage()
    if (image!=null) {
        Log.d("onImageAvailable", "display #${imageDisplayedCount} Timestamp #${imageConsumedCount}: ${image?.timestamp}")
        imageConsumedCount++
        val next_image = reader?.acquireNextImage()
        while (next_image != null) {
            imageConsumedCount++
            image.close()
            image = next_image
            next_image = reader?.acquireNextImage()
        }
        synchronized(uiUpdate) {
            runOnUiThread(uiUpdate)
            (uiUpdate as java.lang.Object).wait()
        }

        imageDisplayedCount++

        val buffer = image?.planes?.get(0)?.buffer
        val bytes = ByteArray(buffer?.capacity()!!)
        buffer.rewind()
        buffer.get(bytes)
        SaveImageTask().execute(bytes)
    }

    image?.close()
}

As an afterthought, I don't like this wait() in the camera callback thread, and I don't think it is really needed. But most likely, it doesn't really matter, because the actual acquisition happens out-of-process, and the paused ImageReader does not effect the burst.

Update Looking at the log that you gathered from the onImageAvailable() callback:

Did you set maxImages to 3? I expected the displayed counter and consumed counter to become different…

The delays between image timestamps are not unifrom, rounded to milliseconds:

33 33 433 200 100 67 167 100 100

which shows that first three frames came without wait for UI to get updated. You could drop the two first frames based on this criterion alone.

As for onCaptureStarted(), the docs are a bot vague (could be on purpose).

This method is called when the camera device has started capturing the output image for the request, at the beginning of image exposure, or when the camera device has started processing an input image for a reprocess request.

Is this or saying that the method may be called 3 times for one capture session? I don't know.

What is explicitly documented, though, is that the timestamps of images are aligned with the timestamp received in onCaptureStarted(). This provides another criterion to filter out the images that were gathered too early.

Actually, I would suggest to log same timestamps from call capture method every time I capture new image in my OnImageAvailableListener listener. It is very possible that this will let your application to capture images in a more predictable way, but in same time.

If you look at the timestamps of the log messages (as reported by logcat, in milliseconds; they may not be aligned with the image timestamps), this can help you understand the timeline even better.

Thank you very much for the reply! All the things you have said makes perfect sense to me, but unfortunately, setting the `maxImages` to 3 and using `Image.Reader.acquireLatestImage()` did not help. I even tried to use an `AtomicInteger` for dropping first few (2) images, but it did not help. Every time the first 3 picures are same (showing number 0). — vp24, Apr 22 '19 at 21:29
I don't exactly understand how you use `AtomicInteger` for dropping first images. — Alex Cohn, Apr 23 '19 at 07:25
In `onImageAvailable` after checking if the image is null, I use `incrementAndGet()` on my `AtomicInteger` (initially set to 0) to check if it is greater than 2 (i have to drop first 2 images). If it is greater than two, I just continue to the same code for saving image as before. I am using `AtomicInteger` because I wanto it to be thread-safe. Is this a bad way for dropping images? — vp24, Apr 23 '19 at 08:41
I have proposed some way to handle the situation. I am curious what your logs will show (the devices I work with these days, do not support camera2 burst, unfortunately). — Alex Cohn, Apr 23 '19 at 09:37
You don't really need atomics because all camera callbacks are guaranteed to come on the same (handler) thread. — Alex Cohn, Apr 23 '19 at 09:38
I edited my question with proposed logs. I still haven't managed to get this working — vp24, Apr 23 '19 at 18:46
Another thing I have noticed: I have put logging to `CameraCaptureSession.CaptureCallback` in the method `onCaptureStarted`. What i have found out is that the application every time enters the `onCaptureStarted` method 3 times and then afterwards it is entering the `onImageAvailable` method. If I am understanding this right, that is why the first 3 images are the same: their capture have started before entering the first `onImageAvailable`, but they just haven't been saved i.e. the display is still showing number 0. Any idea to solve this problem? — vp24, Apr 23 '19 at 19:14
I had managed to get this to work, but it is still a bit slow and I would say ineffective. Check my updated question please. — vp24, Apr 24 '19 at 15:28

Android Camera2: Update UI between image captures with captureBurst is not synchronized

1 Answers1