5

I'm using the Firebase-ML barcode decoder in a streaming (live) fashion using the Camera2 API. The way I do it is to set up an ImageReader that periodically gives me Images. The full image is the resolution of my camera, so it's big - it's a 12MP camera.

The barcode scanner takes about 0.41 seconds to process an image on a Samsung S7 Edge, so I set up the ImageReaderListener to decode one at a time and throw away any subsequent frames until the decoder is complete.

The image format I'm using is YUV_420_888 because that's what the documentation recommends, and because if you try to feed the ML Barcode decoder anything else it complains (run time message to debug log).

All this is working but I think if I could crop the image it would work better. I'd like to leave the camera resolution the same (so that I can display a wide SurfaceView to help the user align his camera to the barcode) but I want to give Firebase a cropped version (basically a center rectangle). By "work better" I mean mostly "faster" but I'd also like to eliminate distractions (especially other barcodes that might be on the edge of the image).

This got me trying to figure out the best way to crop a YUV image, and I was surprised to find very little help. Most of the examples I have found online do a multi step process where you first convert the YUV image into a JPEG, then render the JPEG into a Bitmap, then scale that. This has a couple of problems in my mind

  • It seems like that would have significant performance implications (in real time). This would help me accomplish a few things including reducing some power consumption, improving the response time, and allowing me to return Images to the ImageReader via image.close() more quickly.
  • This approach doesn't get you back to Image, so you have to feed firebase a Bitmap instead, and that doesn't seem to work as well. I don't know what firebase is doing internally but I kind of suspect it's working mostly (maybe entirely) off of the Y plane and that the translation if Image -> JPEG -> Bitmap muddies that up.

I've looked around for YUV libraries that might help. There is something in the wild called libyuv-android but it doesn't work exactly in the format firebase-ml wants, and it's a bunch of JNI which gives me cross-platform concerns.

I'm wondering if anybody else has thought about this and come up with a better solution for croppying YUV_420_488 images in Android. Am I not able to find this because it's a relatively trivial operation? There's stride and padding to be concerned with among other things. I'm not an image/color expert, and i kind of feel like I shouldn't attempt this myself, my particular concern being I figure out something that works on my device but not others.

Update: this may actually be kind of moot. As an experiment I looked at the Image that comes back from ImageReader. It's an instance of ImageReader.SurfaceImage which is a private (to ImageReader) class. It also has a bunch of native tie-ins. So it's possible that the only choice is to do the compress/decompress method, which seems lame. The only other thing I can think of is to make the decision myself to only use the Y plane and make a bitmap from that, and see if Firebase-ML is OK with that. That approach still seems risky to me.

wz2b
  • 1,017
  • 7
  • 24
  • See this related discussion: *[How to manipulate on the fly YUV Camera frame efficiently in Android?](https://stackoverflow.com/questions/54025913)*. At any rate, you probably looked at a wrong ImageReader, the one that controls display, not the one that passes the YUV image to barcode scanner. – Alex Cohn Jan 09 '19 at 20:18
  • That sort of helps - at least it shows me something similar. I now think I could probably crop the image but I have another problem: android.media.Image is abstract but its constructor is package protected. So I can't subclass Image. That means I can't even make my own Image (at least not by simple subclassing) – wz2b Jan 14 '19 at 16:35
  • You cannot create an `Image`, but why do you need such? You can crop an existing `Image`. – Alex Cohn Jan 14 '19 at 21:30
  • Alex: how? By setting cropRect? Experimentally I've convinced myself that Firebase Vision ignores the cropRect. I tested this by creating a small cropRect then verifying that the barcode scanner still decodes things outside of that Rect. This would be an ideal solution if it worked. – wz2b Jan 15 '19 at 18:27
  • 1
    *Firebase Vision ignores the cropRect:* I believe that this deserves a [new feature request](https://github.com/firebase/firebase-android-sdk/issues/new?template=fr.md), but may even be considered a bug. In the meanwhile, before Google take time to fix this, you can use [FirebaseVisionImage.fromByteArray()](https://firebase.google.com/docs/reference/android/com/google/firebase/ml/vision/common/FirebaseVisionImage.html#fromByteArray(byte[],%20com.google.firebase.ml.vision.common.FirebaseVisionImageMetadata)). The input would be cropped from the **Y** plane of your camera image. – Alex Cohn Jan 16 '19 at 16:32
  • @AlexCohn How can I crop the Y plane than? Before use it in the FirebaseVisionImage.fromByteArray() – GMX Sep 20 '19 at 15:25
  • 1
    @GMX if you have a ByteArray, you can copy part of it to another ByteArray, and pass smaller dimension via forged [metadata](https://firebase.google.com/docs/reference/android/com/google/firebase/ml/vision/common/FirebaseVisionImageMetadata.Builder). The easiest crop would make a new image with same width as the old one. – Alex Cohn Sep 21 '19 at 10:52
  • Thank you @AlexCohn, I'm trying that, do u have some hint on the first part? How to copy only the part that I need – GMX Sep 21 '19 at 12:02
  • check out the answers to these questions; they are using CameraX but still the same image format (YUV_420_888) so should work the same way: https://stackoverflow.com/questions/57222800/android-how-to-crop-images-using-camerax and https://stackoverflow.com/questions/63390243/is-there-a-way-to-crop-image-imageproxy-before-passing-to-mlkits-analyzer – Adam Burley Jun 22 '21 at 11:30

1 Answers1

0

I tried to scale down the YUV_420_888 output image today. I think it quite similar to the cropping image.

In my case, I will put the 3 byte arrays from the Image.plane for representing the Y,U and V. yBytes = Image.plane[0] uBytes = Image.plane[1] vBytes = Image.plane[2]

Then I convert it to the RGB array for bitmap converting. I found that if I read the YUV array with the original Image width and height by step 2 Then I can scale half of my Bitmap Image.

What should you prepare: yBytes, uBytes, vBytes, width of Image, height of Image, y row stride, uv RowStride, uv pixelStride, output array (The length of output array length should be equal to the output image width * height. For me the size is 1/4 original width and height)

It means if you can find the cropping area positions(The four corner of the Image) in your image, you can just fill the new RGB array with the YUV data.

Hope it can help you to solve the problem.

Trify
  • 1
  • 1
  • 1