Camerax image analysis: Convert image to bytearray or ByteBuffer

Question

I did lot of reading tried so many different methods available. CameraX is producing yuv_420_888 format Image object and provides it to the ImageAnalysis.

However, there is no way to convert this to a bytebuffer in order to scale, convert to bitmap and run detection operations. I tried following and numerous other proposed techniques.

Converting ImageProxy to Bitmap

All those created grayscale (even after using all 3 planes) and some overlay color shade image. It also created glitchy outputs in-between frames sometime which I could not figure out a reason.

What’s the proper way to get a simple byte array so that it can be converted to bitmap later?

Also how to get cameraX authors attention?

Answer 1

 fun imageProxyToByteArray(image: ImageProxy): ByteArray {
            val yuvBytes = ByteArray(image.width * (image.height + image.height / 2))
            val yPlane = image.planes[0].buffer
            val uPlane = image.planes[1].buffer
            val vPlane = image.planes[2].buffer

            yPlane.get(yuvBytes, 0, image.width * image.height)

            val chromaRowStride = image.planes[1].rowStride
            val chromaRowPadding = chromaRowStride - image.width / 2

            var offset = image.width * image.height
            if (chromaRowPadding == 0) {

                uPlane.get(yuvBytes, offset, image.width * image.height / 4)
                offset += image.width * image.height / 4
                vPlane.get(yuvBytes, offset, image.width * image.height / 4)
            } else {
                for (i in 0 until image.height / 2) {
                    uPlane.get(yuvBytes, offset, image.width / 2)
                    offset += image.width / 2
                    if (i < image.height / 2 - 2) {
                        uPlane.position(uPlane.position() + chromaRowPadding)
                    }
                }
                for (i in 0 until image.height / 2) {
                    vPlane.get(yuvBytes, offset, image.width / 2)
                    offset += image.width / 2
                    if (i < image.height / 2 - 1) {
                        vPlane.position(vPlane.position() + chromaRowPadding)
                    }
                }
            }

            return yuvBytes
        }

Answer 2

You can use this class ripped from Mlkit Pose Detection.

Mlkit pose detection: BitmapUtils.java

object ImageProxyUtils {

fun getByteArray(image: ImageProxy): ByteArray? {
    image.image?.let {
        val nv21Buffer = yuv420ThreePlanesToNV21(
            it.planes, image.width, image.height
        )

        return ByteArray(nv21Buffer.remaining()).apply {
            nv21Buffer.get(this)
        }
    }

    return null
}

private fun yuv420ThreePlanesToNV21(
    yuv420888planes: Array<Plane>,
    width: Int,
    height: Int
): ByteBuffer {
    val imageSize = width * height
    val out = ByteArray(imageSize + 2 * (imageSize / 4))
    if (areUVPlanesNV21(yuv420888planes, width, height)) {

        yuv420888planes[0].buffer[out, 0, imageSize]
        val uBuffer = yuv420888planes[1].buffer
        val vBuffer = yuv420888planes[2].buffer
        vBuffer[out, imageSize, 1]
        uBuffer[out, imageSize + 1, 2 * imageSize / 4 - 1]
    } else {
        unpackPlane(yuv420888planes[0], width, height, out, 0, 1)
        unpackPlane(yuv420888planes[1], width, height, out, imageSize + 1, 2)
        unpackPlane(yuv420888planes[2], width, height, out, imageSize, 2)
    }
    return ByteBuffer.wrap(out)
}

private fun areUVPlanesNV21(planes: Array<Plane>, width: Int, height: Int): Boolean {
    val imageSize = width * height
    val uBuffer = planes[1].buffer
    val vBuffer = planes[2].buffer

    val vBufferPosition = vBuffer.position()
    val uBufferLimit = uBuffer.limit()

    vBuffer.position(vBufferPosition + 1)
    uBuffer.limit(uBufferLimit - 1)

    val areNV21 =
        vBuffer.remaining() == 2 * imageSize / 4 - 2 && vBuffer.compareTo(uBuffer) == 0

    vBuffer.position(vBufferPosition)
    uBuffer.limit(uBufferLimit)
    return areNV21
}

private fun unpackPlane(
    plane: Plane,
    width: Int,
    height: Int,
    out: ByteArray,
    offset: Int,
    pixelStride: Int
) {
    val buffer = plane.buffer
    buffer.rewind()
    val numRow = (buffer.limit() + plane.rowStride - 1) / plane.rowStride
    if (numRow == 0) {
        return
    }
    val scaleFactor = height / numRow
    val numCol = width / scaleFactor

    var outputPos = offset
    var rowStart = 0
    for (row in 0 until numRow) {
        var inputPos = rowStart
        for (col in 0 until numCol) {
            out[outputPos] = buffer[inputPos]
            outputPos += pixelStride
            inputPos += plane.pixelStride
        }
        rowStart += plane.rowStride
    }
}

}

Answer 3

You just need to use imageProxy.image?.toBitmap() to convert imageProxy and then convert bitmap to bytearray as follow:

Here's an example:

private fun takePhoto() {

    camera_capture_button.isEnabled = false

    // Get a stable reference of the modifiable image capture use case
    val imageCapture = imageCapture ?: return

    imageCapture.takePicture(
        ContextCompat.getMainExecutor(this),
        object : ImageCapture.OnImageCapturedCallback() {

            @SuppressLint("UnsafeExperimentalUsageError")
            override fun onCaptureSuccess(imageProxy: ImageProxy) {
                val bitmapImage = imageProxy.image?.toBitmap()
                   
                val stream = ByteArrayOutputStream()
                bitmapImage.compress(Bitmap.CompressFormat.PNG, 90, stream)
                val image = stream.toByteArray()
                    
                
            }

            override fun onError(exception: ImageCaptureException) {
                super.onError(exception)
            }
        })
}