1

I am using libjingle_peerconnection.jar for video calling based on this project and everything is fine.

Now I need to detect face in real time and also detect smile. I want to use ML Kit for this. Therefore, I need to access the byte buffer of each frame.

As Google says:

If you use the Camera2 API, capture images in ImageFormat.YUV_420_888 format. If you use the older Camera API, capture images in ImageFormat.NV21 format.

To access the frame, I copied and pasted the SurfaceViewRenderer source into my project and used it. In this class, there is a method called renderFrame that takes a org.webrtc.VideoRenderer.I420Frame as a parameter.

There are two problems.

  1. When the camera is used, the yuvPlanes value of the frame is null.
  2. When remote is used, the yuvPlanes value is not null, but I can't convert it to YUV_420_888 and NV21 frame. How should I do this?

I420Frame class properties:

    public static class I420Frame {
        public final int width;
        public final int height;
        public final int[] yuvStrides;
        public ByteBuffer[] yuvPlanes;
        public final boolean yuvFrame;
        public final float[] samplingMatrix;
        public int textureId;
        private long nativeFramePointer;
        public int rotationDegree;
    }
Hussein Yaqoobi
  • 442
  • 5
  • 20
  • I think you can use libyuv from Google to do it, it has lots of methods to convert frames from one type to another, but most likely you will get some delay due to frames conversion – easy_breezy May 01 '23 at 12:05
  • I found this library: https://github.com/crow-misia/libyuv-android. Any tips on how to use it? – Hussein Yaqoobi May 02 '23 at 09:50

0 Answers0