Get ByteBuffer from Image for TensorFlow Lite Model

Question

I am creating an android app to run on Google Glass Enterprise Edition 2 that does Real-time Face Recognition. I am using Camera X as my Camera API and TensorFlow Lite (TFLite) as my classification model. However, the TFLite model input requires ByteBuffer which I am unable to convert into from the image retrieved from CameraX.

How do I get my Image from CameraX into ByteBuffer class for my TFLite Model?

Camera X Image Analysis: Reference

            val imageAnalysis = ImageAnalysis.Builder()
                    .setTargetResolution(Size(640, 360))
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                    .build()

            imageAnalysis.setAnalyzer(AsyncTask.THREAD_POOL_EXECUTOR, ImageAnalysis.Analyzer { imageProxy ->
                val rotationDegrees = imageProxy.imageInfo.rotationDegrees
                val mediaImage = imageProxy.image

                if (mediaImage != null) {
                    val image = InputImage.fromMediaImage(mediaImage, rotationDegrees)

                    /* Classify the Image using TensorFlow Lite Model */

                }

            })

TensorFlow Model Sample Code

val model = FaceRecognitionModel.newInstance(context)

// Creates inputs for reference.
val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
inputFeature0.loadBuffer(byteBuffer)

// Runs model inference and gets result.
val outputs = model.process(inputFeature0)
val outputFeature0 = outputs.outputFeature0AsTensorBuffer

// Releases model resources if no longer used.
model.close()

There is also an option to use ML kit and face recognition. With this option you can use immediately the media.Image from the camera. Check this https://developers.google.com/ml-kit/vision/face-detection/android#using-a-media.image — Farmaker, Sep 21 '21 at 04:27
@Farmaker, I did use [media.Image](https://developers.google.com/ml-kit/vision/face-detection/android#using-a-media.image) from my camera as shared in my code `val mediaImage = imageProxy.image` above. — Kenneth Kwek, Sep 21 '21 at 11:40

score 2 · Answer 1 · answered Feb 17 '21 at 01:23

Try using the TensorImage class from the TensorFlow Lite Support Library.

Roughly, you can follow these steps.

Convert the Image object into Bitmap. There should be other Stackoverflow questions on how to do this (e.g., this answer)
Create a TensorImage object from the Bitmap object using TensorImage.fromBitmap() factory.
Call getBuffer() method on the TensorImage object to get the underlying ByteBuffer.

You might also want to do some image pre-processing, in case the image from CameraX doesn't exactly match the format expected by the model. For this, you can explore the ImageProcessor utility.

score 0 · Answer 2 · answered Sep 20 '21 at 16:54

I have made some findings and I applied these to help shorten my problem.

The image I have gotten from CameraX is in YUV. I have trained my model in RGB in 224X224. To suit my issue, I first convert the image to RGB Bitmap, then crop it into 224X224. Afterwards convert Bitmap to ByteBuffer.
As for my TFLite Model, the TFLite model managed to accept the converted RGB ByteBuffer image and process it. Returning back TensorBuffer.

Get ByteBuffer from Image for TensorFlow Lite Model

2 Answers2