Android camera2 ImageReader causing freeze

Question

I have been using camera2 in an opengl application to write to a SurfaceTexture. Since this is going to put the image in the wrong format, I decided to use an ImageReader object to capture the camera output in : ImageFormat.YUV_420_888 as recommended by MLKit.

When I added this, things seem to work okay, but when the FPS of my application drops for a second, the camera stops updating (only happens when I use the ImageReader).

I have seen posts like : Camera2 ImageReader freezes repeating capture request

however, I have

        ImageReader reader = ImageReader.newInstance(100, 100,
                ImageFormat.YUV_420_888, 3);
        reader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
            @Override
            public void onImageAvailable(ImageReader reader) {
                Log.d("FINDME", " ON AVAILABLE STARTED");


                Image image = null;
                try {
                    image = reader.acquireNextImage();
                    //Do whatever you want with your Image
                    if (image != null) {
                        image.close();
                    }
                } catch (IllegalStateException iae) {
                    if (image != null) {
                        image.close();
                    }
                }


                Log.d("FINDME", " ON AVAILABLE DONE");
            }
        }, handler2);

        Surface previewSurface = new Surface(mSurfaceTexture);
        mSurfaces.add(previewSurface);
        mSurfaces.add(reader.getSurface());
        mPreviewBuilder.addTarget(previewSurface);
        mPreviewBuilder.addTarget(reader.getSurface());

        mCameraDevice.createCaptureSession(mSurfaces, mCaptureSessionStateCallback,
                mBackgroundHandler);

and doesn't seem to help. It does run fine if the FPS stays high, but if I try to swap to different rendering features (Shaders), the fps can drop for a split second, which freezes the camera. The app still runs fine, just the camera source no longer updates and never recovers. It isn't all shaders that cause this as well, only the ones that are more involved.

in logcat I see:

12-12 22:50:43.824 21329 23300 D FINDME  :  ON AVAILABLE DONE
12-12 22:50:43.854 21329 21371 I MediaRecorderJNI: finalize
12-12 22:50:43.854 21329 21371 I MediaRecorderJNI: release
12-12 22:50:43.854  1015 17804 D MPEG4Writer: Video track stopping. Stop source
12-12 22:50:43.854  1015 17804 E MPEG4Writer: Stop() called but track is not started
12-12 22:50:43.854  1015 17804 D MPEG4Writer: Audio track stopping. Stop source
12-12 22:50:43.854  1015 17804 E MPEG4Writer: Stop() called but track is not started
12-12 22:50:43.855  1015 22459 I ACodec  : [OMX.SEC.naac.enc] Now Executing->Idle
12-12 22:50:43.855  1090 22460 I SEC_BASE_COMP: SEC_OMX_ComponentStateSet: Cur(3) -> Dest(2)
12-12 22:50:43.857  1015 22459 I ACodec  : [OMX.SEC.naac.enc] Now Idle->Loaded
12-12 22:50:43.860  1090 22460 I SEC_BASE_COMP: SEC_OMX_ComponentStateSet: Cur(2) -> Dest(1)
12-12 22:50:43.861  1015 22459 I ACodec  : [OMX.SEC.naac.enc] Now Loaded
12-12 22:50:43.861  1090 25268 I SEC_BASE_COMP: [OMX.SEC.naac.enc]: Normally terminated
12-12 22:50:43.862  1015 22459 I ACodec  :  [OMX.SEC.naac.enc] Now uninitialized
12-12 22:50:43.862  1015 22459 I ACodec  :  [] Now kWhatShutdownCompleted event : 8544
12-12 22:50:43.862  1015 22459 I MediaCodec: Codec shutdown complete
12-12 22:50:43.865  1015 17804 I StagefrightRecorder: property set(service.camera.rec.running, 0) result = 0
12-12 22:50:43.866  1015 22451 I ACodec  : [OMX.qcom.video.encoder.avc] Now Executing->Idle
12-12 22:50:43.883  1015 22451 I ACodec  : [OMX.qcom.video.encoder.avc] Now Idle->Loaded
12-12 22:50:43.883  1015 22451 I ACodec  : [OMX.qcom.video.encoder.avc] Now Loaded
12-12 22:50:43.891  1090  5365 I OMX-VENC: Component Deinit
12-12 22:50:43.891  1090  5365 I OMX-VENC: Video encode perflock released
12-12 22:50:43.892  1015 22451 I ACodec  :  [OMX.qcom.video.encoder.avc] Now uninitialized
12-12 22:50:43.892  1015 22451 I ACodec  :  [] Now kWhatShutdownCompleted event : 8544
12-12 22:50:43.893  1015 22450 I MediaCodec: Codec shutdown complete
12-12 22:50:43.893  1015 22450 I ResourceManagerService: MediaInfo removed 2397600 (width 1080 height 2220) remained 15297120 
12-12 22:50:43.893  1015 22450 I ResourceManagerService: getMediaResourceInfo resourceType : 2, size : 1
12-12 22:50:43.893  1015 22450 I ResourceManagerService: writeResourceInfo (PID : 21329, clientID : 3989906560, non-secure-codec/video-codec:1, 1080x2220(fps:30) - SoftCodec : No, Encorder : Yes)
12-12 22:50:43.893 28011 28024 V ResourceManagerHelper-JNI: JNIMediaResourceHelper::notify eventType : 2, ext1 : 0, ext2 : 0
12-12 22:50:43.893 28011 28024 V ResourceManagerHelper-JNI: notify eventType : 2, ext1 : 0, ext2 : 0
12-12 22:50:43.893  3381 29213 V ResourceManagerHelper-JNI: JNIMediaResourceHelper::notify eventType : 2, ext1 : 0, ext2 : 0
12-12 22:50:43.893  3381 29213 V ResourceManagerHelper-JNI: notify eventType : 2, ext1 : 0, ext2 : 0
12-12 22:50:43.893 28011 28024 I SemMediaResourceHelper: makeMediaResourceInfo mOwnResourceEventExcluded : false, mPid : 28011
12-12 22:50:43.893 28011 28024 I SemMediaResourceHelper: [1] makeMediaResourceInfo resourceType : 2 isSecured : false, pid : 21329, client id : 3989906560
12-12 22:50:43.893  3381 29213 I SemMediaResourceHelper: makeMediaResourceInfo mOwnResourceEventExcluded : false, mPid : 3381
12-12 22:50:43.893  3381 29213 I SemMediaResourceHelper: [1] makeMediaResourceInfo resourceType : 2 isSecured : false, pid : 21329, client id : 3989906560
12-12 22:50:43.893 28011 28047 I SemMediaResourceHelper: onRemove
12-12 22:50:43.893  3381  3381 I SemMediaResourceHelper: onRemove
12-12 22:50:43.893 28011 28047 I DrmMediaResourceHelper: onRemove size = 1
12-12 22:50:43.893 28011 28047 E DrmMediaResourceHelper: onRemove making Dpdrm to 0 second point 
12-12 22:50:43.895   691 11810 V APM_AudioPolicyManager: releaseInput() 1630
12-12 22:50:43.895   691 11810 V APM_AudioPolicyManager: closeInput(1630)
12-12 22:50:43.896 28011 28047 E DrmManagerClient: DRM_DISPLAYPORT_DISABLE failed
12-12 22:50:43.896  1015 17804 V MediaPlayerService: Delete media recorder client
12-12 22:50:43.897   691 11810 D MultiRecordManager: closeInputStream() moduleHandle:10, modulename:primary, stream_in:0xeab9f0d0, source:0, dev:0xeab95fb0
12-12 22:50:43.897   691 11810 I MultiRecordManager: closeInputStream() skipMultiRecord
12-12 22:50:43.897 21329 21371 W System  : A resource failed to call release. 
12-12 22:50:43.898   691 11810 V APM_AudioPolicyManager: releaseInput() exit
12-12 22:50:43.898 21329 22463 E FINDME: VideoFragment: onClosed: android.hardware.camera2.impl.CameraCaptureSessionImpl@83f26d4
12-12 22:50:43.898   691 30849 V APM_AudioPolicyManager: listAudioPorts() role 0 type 0 num_ports 0 ports 0xeab97700
12-12 22:50:43.898   691 30849 V APM_AudioPolicyManager: listAudioPorts() got 0 ports needed 11
12-12 22:50:43.899   691 30849 V APM_AudioPolicyManager: listAudioPorts() role 0 type 0 num_ports 11 ports 0xe70d8d80
12-12 22:50:43.899   691 30849 V APM_AudioPolicyManager: listAudioPorts() got 11 ports needed 11
12-12 22:50:43.899   653  1131 D audio_hw_primary: adev_close_input_stream: enter:stream_handle(0xe885a500)
12-12 22:50:43.899   653  1131 V audio_hw_sec: sec_stop_voice_note_recording
12-12 22:50:43.899   653  1131 D audio_hw_primary: in_standby: enter: stream (0xe885a500) usecase(18: audio-record)
12-12 22:50:43.900   691 11810 V APM_AudioPolicyManager: listAudioPorts() role 0 type 0 num_ports 0 ports 0xeab976a8
12-12 22:50:43.900   691 11810 V APM_AudioPolicyManager: listAudioPorts() got 0 ports needed 11
12-12 22:50:43.900   691 11810 V APM_AudioPolicyManager: listAudioPorts() role 0 type 0 num_ports 11 ports 0xe7093580
12-12 22:50:43.900   691 11810 V APM_AudioPolicyManager: listAudioPorts() got 11 ports needed 11
12-12 22:50:43.908 21329 21376 E BufferQueueProducer: [ImageReader-1280x720f23m3-21329-3] queueBuffer: BufferQueue has been abandoned
12-12 22:50:43.908   982 23334 E Surface : queueBuffer: error queuing buffer to SurfaceTexture, -19
12-12 22:50:43.908   982 23334 E Camera3-OutputStream: returnBufferCheckedLocked: Stream 1: Error queueing buffer to native window: No such device (-19)
12-12 22:50:43.908   982 23334 E Camera3-Device: Can't return buffer to its stream: No such device (-19)
12-12 22:50:43.940 21329 21376 E BufferQueueProducer: [ImageReader-1280x720f23m3-21329-3] dequeueBuffer: BufferQueue has been abandoned
12-12 22:50:43.940   982 23322 E Camera3-OutputStream: getBufferLockedCommon: Stream 1: Can't dequeue next output buffer: No such device (-19)
12-12 22:50:43.940   982 23322 E Camera3-Device: RequestThread: Can't get output buffer, skipping request: No such device (-19)
12-12 22:50:43.949 21329 21376 E BufferQueueProducer: [ImageReader-1280x720f23m3-21329-3] queueBuffer: BufferQueue has been abandoned
12-12 22:50:43.950   982 23334 E Surface : queueBuffer: error queuing buffer to SurfaceTexture, -19

Is the listener not called again or is it hang on `reader.acquireNextImage()` because of the FPS drop? If the latter is the case, you can try to use `reader.acquireLastImage()` which *should* fetch the latest image on camera. — Furkan Yurdakul, Dec 13 '19 at 06:27
Its not being called again. I have verified that the last call to the listener completed. I also did try acquireLastImage and didn't make a difference. — Luple, Dec 13 '19 at 06:31
Hmm. I have noticed that you are passing `null` on the image listener, which seems to indicate that the code is running on UI thread, if you're not calling it from a background handler, that is. Maybe try to create a background thread (with a looper, which can be `HandlerThread`), use the listener there, and then add the shader and display it on the surface on UI thread. That way the thread associated with the camera should not stop, hence the listener should be stable. — Furkan Yurdakul, Dec 13 '19 at 06:34
I have updated so createCaptureSession uses handler1, and the listener runs on handler2, and it still crashed. it seems like the issue is that the surfacetexture target is blocked, while the imagereader keeps running and thinks it is no longer being updated so it self destructs. But I have no basis to go off of besides the posted log. updated post with some more code. — Luple, Dec 13 '19 at 06:41
Crash seems to be different from the image reader releasing the instance, do you have actual stack traces of the crash? If so, could you post it in the question? Maybe we can help from there. — Furkan Yurdakul, Dec 13 '19 at 06:50
I am not getting a crash, so no stack trace is coming up. I have put in a lot more of the log from the last time the listener was called. — Luple, Dec 13 '19 at 06:53
Not sure, but the 2nd and 3rd line of the logcat you shared indicates that garbage collector has run, which can also lead a FPS drop and shutting down the camera instance. So either, there is some sort of a memory leak in your code, or the device is not capable of doing it, that's the only things that I can think of at this point. — Furkan Yurdakul, Dec 13 '19 at 06:56
Thanks for all of the help. I will play around with my architecture and see if I can avoid it all together. I will see if I can keep the camera off the same thread as my opengl code, which is where I think the issue is coming from. — Luple, Dec 13 '19 at 07:09
Resolved. ImageReader reader was falling out of scope, causing the buffer to disappear after a while, causing it to randomly stop. — Luple, Dec 16 '19 at 06:25

Android camera2 ImageReader causing freeze

0 Answers0