How to get Bitmap from session.update() in ARCore [Android Studio]

Question

I am trying to get a Bitmap from the current frame of my ARSession with ARCore. But it always equals null. I've already been searching the web for quite a while but cannot figure out what I am doing wrong.

try {
    capturedImage = mFrame.acquireCameraImage();

    ByteBuffer buffer = capturedImage.getPlanes()[0].getBuffer();

    byte[] bytes = new byte[buffer.capacity()];

    Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length,null);

    if (bitmap == null) 
        Log.e(TAG,"Bitmap was NOT initialized!");

} catch(Exception e){

}

I am getting mFrame from onDrawFrame of my GLSurfaceView which I use to display the camera image. Everything works just fine except that my Bitmap equals null.

I am using a Button, so that only a single Frame is being used, as follows:

scanButton = (Button) findViewById(R.id.scanButton);
scanButton.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        checkbox = false;
        if (capturedImage!=null) capturedImage.close();
            BitmapMethod();
    }
});

capturedImage, buffer and bytes all do not equal null.

Is there probably something wrong with mFrame.acquireCameraImage()?

Thanks a lot

Answer 1

Is there probably something wrong with mFrame.acquireCameraImage()?

No, mFrame.acquireCameraImage() works as intended.

But it always equals null

The Bitmap will always equal null since bitmap factory does not understand the image data that is passed to it.

The method mFrame.acquireCameraImage() responds with an object of type Image that is in the YUV format or YCbCr. These types of images have 3 planes which is explained here very nicely. The ByteArray contained in these planes may be read directly by a CPU/GPU in native code. BitmapFactory cannot read this type of data. Hence, you need to convert this YUV image into something else.

For that, you need to use YuvImage class to create an instance of YUV & then convert it into JPEG using the compressToJpeg method. Once you have the byteArray from this, you can simply do what you're doing above. Use BitmapFactory to convert it into Bitmap and add it to your ImageView.

Note : YUV has 3 planes. Create a single bytearray from all planes & then pass it to YUV constructor. Though not elaborate, it should look something similar to this :

//The camera image received is in YUV YCbCr Format. Get buffers for each of the planes and use them to create a new bytearray defined by the size of all three buffers combined
val cameraPlaneY = cameraImage.planes[0].buffer
val cameraPlaneU = cameraImage.planes[1].buffer
val cameraPlaneV = cameraImage.planes[2].buffer

//Use the buffers to create a new byteArray that 
val compositeByteArray = ByteArray(cameraPlaneY.capacity() + cameraPlaneU.capacity() + cameraPlaneV.capacity())

cameraPlaneY.get(compositeByteArray, 0, cameraPlaneY.capacity())
cameraPlaneU.get(compositeByteArray, cameraPlaneY.capacity(), cameraPlaneU.capacity())
cameraPlaneV.get(compositeByteArray, cameraPlaneY.capacity() + cameraPlaneU.capacity(), cameraPlaneV.capacity())

val baOutputStream = ByteArrayOutputStream()
val yuvImage: YuvImage = YuvImage(compositeByteArray, ImageFormat.NV21, cameraImage.width, cameraImage.height, null)
yuvImage.compressToJpeg(Rect(0, 0, cameraImage.width, cameraImage.height), 75, baOutputStream)
val byteForBitmap = baOutputStream.toByteArray()
val bitmap = BitmapFactory.decodeByteArray(byteForBitmap, 0, byteForBitmap.size)
imageView.setImageBitmap(bitmap)

That's just a rough code. It has scope for improvement perhaps. Also refer here.

Answer 2

I also winded up recreating the same situation as you were facing. I was getting the Image object as Null.

After some research, I found the problem was in the flow of the logic.

I then wrote the following code and it solved my issue:

I defined the following boolean to be set to capture the current frame on the button click:

private static boolean captureCurrentFrame = false;

This code I wrote in the onClick() function to get the current frame's RGB and Depth Image:

public void captureFrame(View view){
    Toast.makeText(getApplicationContext(),"Capturing depth and rgb photo",Toast.LENGTH_SHORT).show();
    captureCurrentFrame = true;
  }

This section I wrote in onDrawFrame() method just after getting the frame from session.update():

if(captureCurrentFrame) {
        RGBImage = frame.acquireCameraImage();
        DepthImage = frame.acquireDepthImage();

        Log.d("Image","Format of the RGB Image: " + RGBImage.getFormat());
        Log.d("Image","Format of the Depth Image: " + DepthImage.getFormat());

        RGBImage.close();
        DepthImage.close();

        captureCurrentFrame = false;
      }

Reason for getting Null in my case was the code in the onClick listener was getting triggered before going through the onDraw() method, as a result of which the Images were not stored in the variables.

Therefore, I shifted the logic to onDraw() and triggered that flow through the boolean variable that is set by the listener.

Answer 3

I don't know if there is anybody still looking for the answer, but this is my code.

        Image image = mFrame.acquireCameraImage();
        byte[] nv21;
        // Get the three planes.
        ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
        ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
        ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();

        int ySize = yBuffer.remaining();
        int uSize = uBuffer.remaining();
        int vSize = vBuffer.remaining();


        nv21 = new byte[ySize + uSize + vSize];

        //U and V are swapped
        yBuffer.get(nv21, 0, ySize);
        vBuffer.get(nv21, ySize, vSize);
        uBuffer.get(nv21, ySize + vSize, uSize);

        int width = image.getWidth();
        int height = image.getHeight();

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        YuvImage yuv = new YuvImage(nv21, ImageFormat.NV21, width, height, null);
        yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);
        byte[] byteArray = out.toByteArray();
        Bitmap bitmap = BitmapFactory.decodeByteArray(byteArray, 0, byteArray.length);