4

The image quality and the framerate I get when using the camera2 API does not match the one I get when I manually record a video using the camera app to a file.

I am trying to do real-time image processing using OpenCV on Android. I have manually recorded a video using the built-in camera application and everything worked perfectly: the image quality was good, the framerate was a stable 30 FPS.

My min SDK version is 22, so I am using the camera2 API's repeating requests. I have set it up, together with an ImageReader and the YUV_420_888 format. I have tried both the PREVIEW and the RECORD capture request templates, tried manually setting 18 capture request parameters in the builder (eg. disabling auto-white-balance, setting the color correction mode to fast), but the FPS was still around 8-9 and the image quality was poor as well. Another phone yielded the same results, despite its max. FPS being 16.67 (instead of 30).

The culprit is not my image processing (which happens in another thread, except for reading the image's buffer): I checked the FPS when I don't do anything with the frame (I didn't even display the image), it was still around 8-9.

You can see the relevant code for that here:

//constructor:
HandlerThread thread = new HandlerThread("MyApp:CameraCallbacks", Process.THREAD_PRIORITY_MORE_FAVORABLE);
thread.start();
captureCallbackHandler = new Handler(thread.getLooper());

//some UI event:
cameraManager.openCamera(cameraId, new CameraStateCallback()), null);

//CameraStateCallback#onOpened:
//size is 1280x720, same as the manually captured video's
imageReader = ImageReader.newInstance(size.getWidth(), size.getHeight(), ImageFormat.YUV_420_888, 1);
imageReader.setOnImageAvailableListener(new ImageAvailableListener(), captureCallbackHandler);
camera.createCaptureSession(Collections.singletonList(imageReader.getSurface()), new CaptureStateCallback(), captureCallbackHandler);

//CaptureStateCallback#onConfigured:
CaptureRequest.Builder builder = activeCamera.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);
builder.addTarget(imageReader.getSurface());
//setting the FPS range has no effect: this phone only has one option
session.setRepeatingRequest(builder.build(), null, captureCallbackHandler);

//ImageAvailableListener#onImageAvailable:
long current = System.nanoTime();
deltaTime += (current - last - deltaTime) * 0.1;
Log.d("MyApp", "onImageAvailable FPS: " + (1000000000 / deltaTime));
//prints around 8.7
last = current;
try (Image image = reader.acquireLatestImage()) { }
Jacob G.
  • 28,856
  • 5
  • 62
  • 116
Trigary
  • 366
  • 4
  • 14
  • Is it a LEGACY device? – Alex Cohn Jan 31 '19 at 20:20
  • @AlexCohn The camera with the 16.67 max. FPS is a LEGACY device. I'm afraid I can't check the other one at the moment, but it's certainly a possibility. – Trigary Jan 31 '19 at 20:30
  • 1
    I have seen once and again that camera2 API delivers suboptimal results for legacy cameras. I strongly recommend to use the deprecated Camera API in such situations. – Alex Cohn Jan 31 '19 at 20:36
  • 1
    Thank you for the suggestion, I will try that. I will share the results here once I'm done. – Trigary Jan 31 '19 at 20:40
  • 1
    Don't forget to call Camera.open() [from a background HadlerThread](https://stackoverflow.com/a/19154438/192373) for best performance! – Alex Cohn Jan 31 '19 at 21:03
  • Update: while I wasn't able to get the deprecated camera API working on one of the phones (hopefully that will change), the framerate indeed became considerably higher on the other one: 12.4 at 1280x720 resolution while targeting 15. This target FPS can be reached at lower resolutions, but still: the video captured by the built-in application reached 16.67 FPS at 1280x720. – Trigary Feb 03 '19 at 08:01
  • 1
    It may be impossible to beat the built-in recording app, or the **MediaRecorder**, because they may use a shortcut between camera hardware to avc codec, which passes pixels in shared memory, and involves zero copy. This approach was made 'public' for **camera2** API ([ImageFormat.PRIVATE](https://developer.android.com/reference/android/graphics/ImageFormat.html#PRIVATE)), but alas this does not help for LEGACY devices. – Alex Cohn Feb 03 '19 at 08:31

1 Answers1

0

On Samsung Galaxy J3 (2016), doing Camera.Parameters#setRecordingHint(true) (while using the deprecated camera API) achieves exactly what I wanted: the video quality and the framerate becomes the same as the built-in video recorder's. Unfortunately, it also means that I was unable to modify the resolution, and setting that hint did not achieve this same effect on a Doogee X5 MAX.

Trigary
  • 366
  • 4
  • 14