4

I'm very new to android. I'm trying to use the new Android Camera2 api to build a real time image processing application. My application requires to maintain a good FPS rate as well. Following some examples i managed to do the image processing inside the onImageAvailable(ImageReader reader) method available with ImageReader class. However by doing so, i can only manage to get a frame rate around 5-7 FPS.

I've seen that it is advised to use RenderScript for YUV processing with Android camera2 api. Will using RenderScript gain me higher FPS rates? If so please can someone guide me on how to implement that, as i'm new to android i'm having a hard time grasping concepts of Allocation and RenderScript. Thanks in advance.

Hiddenkeg
  • 89
  • 2
  • 7

2 Answers2

1

I don't know what type of image processing you want to perform. But in case that you are interested only in the intensity of the image (i.e. grayvalue information) you don't need any conversion of the YUV data array (e.g. into jpeg). For an image consisting of n pixels the intensity information is given by the first n bytes of the YUV data array. So, just cut those bytes out of the YUV data array:

 byte[] intensity = new byte[width*height];
 intensity = Arrays.copyOfRange(data, 0, width*height);
Settembrini
  • 1,366
  • 3
  • 20
  • 32
  • Thanks for the reply. I'm interested only in the intensity of the image and i'm using the method you've mentioned. However i'm using android.media.ImageReader at the moment. And my application ends up having low FPS rate around 5-7. I wanted to know whether by using RenderScript will i be able to achieve higher FPS rates. – Hiddenkeg Feb 05 '16 at 03:00
  • I have not studied yet the Camera2, but since my app uses the old Camera I will need to do this soon. With the old Camera I can create preview-callbacks that come at much higher rate than the 5-7 you describe, must be about 20-30 per second. Thus, my guess is that you should work with PREVIEWs, and not with actual pictures. From what I see in SO, the preview mechanism in Camera2 is associated with setRepeatingRequest's CaptureRequest.Builder, see e.g. here http://stackoverflow.com/questions/25462277/camera-preview-image-data-processing-with-android-l-and-camera2-api. – Settembrini Feb 05 '16 at 21:48
  • As with regards to Renderscript (which I use a lot), I am pretty sure that this cannot per se increase the frame rate, that makes no sense. But you can use it for post-processing of Images. But agree, the documentation is pretty non-intuitive, to say the least. – Settembrini Feb 05 '16 at 21:51
  • Thanks you for the informative replies. I was able to gain higher FPS rates compromising the resolution. Low resolutions are sufficient for my application and with the lowest resolution camera gives me around 30 FPS. Now i what i need to do i a method to keep the frame rate fixed. [link](http://stackoverflow.com/questions/35208873/fixed-frame-rate-using-android-camera2-api?noredirect=1#comment58133034_35208873) Thanks for the help. – Hiddenkeg Feb 06 '16 at 06:36
1

In theory, you can get the available fps ranges with this call:

characteristics.get(CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES);

and set the desired fps range here:

mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, bestFPSRange);

So in principle, you should choose a range with the same lower and upper bound, and that should keep your frame rate constant.

HOWEVER, on devices with a LEGACY profile, none of the devices I have tested have been able to achieve 30fps at 1080p (S5, Z3 Compact, Huawei Mate S, and HTC One M9). The only way I was able to achieve that was by using a device (LG G4) that turned out to have a FULL profile.

Renderscript will not buy you anything here if you are going to use it inside the onImageAvailable callback. It appears that getting the image at that point is the bottleneck on LEGACY devices since the new camera2 API simply wraps the old one, and is presumably creating so much overhead that the callback does not occur at 30fps anymore. So if Renderscript is to work, you would need to need to create a Surface and find another way of grabbing the frames off of it.

Here is the kicker though... if you move back the deprecated API, I would almost guarantee 30fps at whatever resolution you want. At least that is what I found on all of the devices I tested....

bremen_matt
  • 6,902
  • 7
  • 42
  • 90
  • This is helpful. I'm trying to do something similar and am only targeting old crappy android tablets. I have no idea what I am doing, as I've never played with the Camera (or Camera2) api before. Is there any chance you have some sample code laying around that I could look at? – Chase Roberts Feb 07 '19 at 03:42