Accessing Image Data Bytes in ARCore

Question

I've created an ARCore Session and attached an OpenGL texture id through the Session#setCameraTextureName method to display my camera data. I'd like to have access to the camera image data bytes displayed on the texture.

ARKit and Tango provide access to the image bytes for each frame but there doesn't seem to be anything that easily provides that in the ARCore API.

Is there any other way I can access the image bytes when using ARCore?

score 5 · Answer 1 · answered Dec 08 '17 at 14:41

Maybe that could help you I wanted to obtain the camera view in a bitmap form. I have tested on Samsung s8.

    int w=1080;
    int h = 2220;
    int b[]=new int[w*(0+h)];
    int bt[]=new int[w*h];
    IntBuffer ib = IntBuffer.wrap(b);
    ib.position(0);
    GLES20.glReadPixels(0, 0, w, h, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, ib);

    for(int i=0, k=0; i<h; i++, k++)
    {//remember, that OpenGL bitmap is incompatible with Android bitmap
        //and so, some correction need.
        for(int j=0; j<w; j++)
        {
            int pix=b[i*w+j];
            int pb=(pix>>16)&0xff;
            int pr=(pix<<16)&0x00ff0000;
            int pix1=(pix&0xff00ff00) | pr | pb;
            bt[(h-k-1)*w+j]=pix1;
        }
    }

    sb=Bitmap.createBitmap(bt, w, h, Bitmap.Config.ARGB_8888);

Ian M · Answer 2 · 2017-08-31T23:20:15.653

2

For the time being, your best bet for accessing image data is probably drawing the texture to a renderbuffer and using glReadPixels into a persistent-mapped pixel unpack buffer. Use a fence sync to detect when the glReadPixels is complete.

Another option is to use a compute shader and write directly to a ~~persistent-mapped~~ SSBO. (Disregard persistent-mapped suggestion. I thought EXT_buffer_storage had broader support)

The later is possibly fewer copies (the renderbuffer pixels may still hit DRAM even if you invalidate it after the glReadPixels), but it's also a less-common code path and incurs render/compute changeovers so I don't have intuition about which approach would be more efficient.

edited Aug 31 '17 at 23:20

answered Aug 31 '17 at 18:04

Ian M

672
4
11

Would this let us get at the YUV data from the camera buffer? – MrAlbean Aug 31 '17 at 18:13
1

No. For now there's no way to access the raw YUV data. The OpenGL sampler will convert to RGB for you. – Ian M Aug 31 '17 at 18:20
@IanM Thanks for the answer. Do you think that there will be an official (and more performant) way for this in the future? After all, Computer Vision is important for AR :-) – PhilLab Aug 31 '17 at 18:28
I can't speak to future plans, but I agree that it's a desirable capability. Unfortunately, my understanding is that current Android platform limitations prevent providing a single buffer that can be used as both a GPU texture and CPU-accessible image, so care must be taken in providing that capability. – Ian M Aug 31 '17 at 18:42
@IanM Thanks for the responses. In that case, is there a way to query the native capture geometry (is it 1280p like ARKit)? We wouldn't want to be wasting work to upscale if it is not needed, but we also wouldn't want to reduce the quality unnecessarily. – MrAlbean Aug 31 '17 at 18:49
1

@MrAlbean For the developer preview the camera image will always be 1920x1080. You might also be able to query it with `glGetTexLevelParameteriv`. I've tested the compute shader approach and it seems to be performant enough for at least proof of concept type work. – Ian M Aug 31 '17 at 23:14

score 2 · Accepted Answer · answered Mar 29 '18 at 20:29

2

As of ARCore v1.1.0, there is an API to access the image bytes for the current frame:

https://developers.google.com/ar/reference/java/com/google/ar/core/Frame.html#acquireCameraImage()

answered Mar 29 '18 at 20:29

MrAlbean

540
1
6
18

Is it just me, or does the image returned seem smaller than it ought to be? I get 640x480, I think, but it (qualitatively) seems like the preview shown on screen is higher resolution than that. – Erhannis May 01 '18 at 01:50
3

That's right. The API to provide CPU access to image bytes returns a 640x480 image, even though the preview displays at 1920x1080. There are a couple issues open on GitHub regarding the image resolution, so an improvement might be in the works: https://github.com/google-ar/arcore-android-sdk/issues/92 https://github.com/google-ar/arcore-android-sdk/issues/153 – MrAlbean May 01 '18 at 17:52
As of v1.28.0, this API is retired. https://github.com/google-ar/arcore-android-sdk/releases – Softlion Nov 29 '21 at 08:33

Accessing Image Data Bytes in ARCore

3 Answers3

Linked