ML barcode scanner from camera preview stream?

Question

I'm trying to make a barcode scanner using ML kit Barcode detector, camera2 API and Kotlin. Concerning camera2 I'm starting from Google sample camera2basic Concerning ML kit Barcode detector, I'm starting from doc : Scan Barcodes with ML Kit on Android

In Camera2BasicFragment / createCameraPreviewSession method, I added

previewRequestBuilder.addTarget(imageReader!!.surface)

so, onImageAvailableListener is called each time an image is available.

In Camera2BasicFragment / setUpCameraOutputs method, i changed ImageReader's ImageFormat.JPEG to ImageFormat YUV420_888, so in onImageAvailableListener, ImageReader gives a YUV image

Then here is my onImageAvailableListener :

   private val onImageAvailableListener = ImageReader.OnImageAvailableListener {
        val metadata = FirebaseVisionImageMetadata.Builder()
            .setWidth(480)   // 480x360 is typically sufficient for
            .setHeight(360)  // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_YV12)
            .setRotation(getRotationCompensation(cameraId, activity as Activity, context!!))
            .build()
        BarcodeReader(it.acquireNextImage(), detector, metadata, mListener).run()
    }

In metadata, 'width' and 'heigth' are as suggested in ML kit doc, 'format' is YV12 to handle YUV format

and Barcode Reader is :

class BarcodeReader (private val image: Image,
                     private val detector: FirebaseVisionBarcodeDetector,
                     private val metadata: FirebaseVisionImageMetadata,
                     private val mListener: IBarcodeScanner) : Runnable {
    override fun run() {
        val visionImage = FirebaseVisionImage.fromByteBuffer(image.planes[0].buffer, metadata)
        detector.detectInImage(visionImage)
            .addOnSuccessListener { barcodes ->
                // Task completed successfully
                // [START_EXCLUDE]
                // [START get_barcodes]
                for (barcode in barcodes) {
                    val bounds = barcode.boundingBox
                    val corners = barcode.cornerPoints

                    val rawValue = barcode.rawValue

                    if (rawValue!=null)
                        mListener.onBarcode(rawValue)
                }
                // [END get_barcodes]
                // [END_EXCLUDE]
            }
            .addOnFailureListener {
                // Task failed with an exception
                // ...
                Log.d("barcode", "null")
            }
        image.close()
    }

The detector.detectInImage enters onSuccessListener but NO BARCODE are detected : barcodes array is always empty.

Can somebody help me please ?

Answer 1

You need to hand over the data of all three planes to the FirebaseVisionImage.fromByteBuffer() function. Your code just hands over the first (Y-plane). The YV12 format uses one buffer (array) that contains the Y-data, followed by the U-data, followed by the V-data.

The Image contains 3 separate buffers for the three values (Y, U, and V), however a real frame needs all three in one buffer (array). Thus you need to create one buffer and copy the contents of the three planes into it, in the right order according to the format (YV12 or NV21), and hand over that buffer (array).

See this Wikipedia article and also this SO question/answer which has some more info regarding conversion and layout of the YV12 and NV21 format. Yet another good source regarding these formats is the VideoLan wiki.

A function may look like this:

override fun onImageAvailable(reader: ImageReader) {
    val image = imageReader?.acquireLatestImage() ?: return
    val planes = image.planes
    if (planes.size >= 3) {
        val y = planes[0].buffer
        val u = planes[1].buffer
        val v = planes[2].buffer
        val ly = y.remaining()
        val lu = u.remaining()
        val lv = v.remaining()

        val dataYUV = ByteArray(ly + lu + lv)
        y.get(dataYUV, 0, ly)
        u.get(dataYUV, ly, lu)
        v.get(dataYUV, ly + lu, lv)

        val metadata = FirebaseVisionImageMetadata.Builder()
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_YV12)
            .setWidth(width)
            .setHeight(height)
            .setRotation(rotation)
            .build()

        detector.detectInImage(FirebaseVisionImage.fromByteArray(dataYUV, metadata))
    }
    image.close()
}

Where width, height and rotation depends on your camera/preview and/or image reader settings.

Answer 2

With ImageReader.OnImageAvailableListener, you can simply use FirebaseVisionImage#fromMediaImage(Image image, int rotation) for ImageFormat YUV420_888

As in the docs:

Note that we only support JPEG / YUV_420_888 formats at this moment. If you are using cloud vision detectors, JPEG format is recommended; if you are using on-device detectors, YUV_420_888 will be more efficient.

Setup ImageReader:

 mImageReader = ImageReader.newInstance(mVideoSize!!.width,
                mVideoSize!!.height,
                ImageFormat.YUV_420_888, 3)
        mImageReader!!.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler)

then:

private val mOnImageAvailableListener = ImageReader.OnImageAvailableListener { imageReader ->
    val image = imageReader.acquireLatestImage()
    try {
        mFaceDetector!!
                .detectInImage(
                        FirebaseVisionImage
                                .fromMediaImage(image))
                .addOnSuccessListener { firebaseVisionFaces ->
                    if (firebaseVisionFaces.size > 0) {
                        Log.d(TAG, "onSuccess: FACE DETECTED")
                    }
                }
        image.close()
    } catch (e: NullPointerException) {
        Log.e(TAG, "onImageAvailable: Invalid image provided for detection", e)
    }
}

Note: I have used face detection, barcode could be used in a similar manner.