In Android Screen Recording - How can I get each frame?

Question

I am using the MediaRecorder and MediaProjection Api for recording the screen in android app. I am not able to get each frame that I can send over a rtmp stream. For publishing over an RTMP stream I am using JAVACV Android library.

For eg - In case of live streaming through camera, we can get each frame in onPreviewFrame() callback. After getting each frame I simply use the FFmpegFrameRecorder of JAVACV library for streaming each frame on to the rtmp url.

How can I achieve the same with screen recording?

Any help would be appreciated here. Thanks in advance!

Have you tried get the key frame buffer from `muxer.writeSampleData(trackIndex, encodedData, info)` assuming you are doing something similar to: https://github.com/google/grafika/blob/master/app/src/main/java/com/android/grafika/ScreenRecordActivity.java (assuming frame shows up frequently enough which it may not for your purposes). — Morrison Chang, Jan 28 '20 at 07:16
@MorrisonChang: Yes I have tried this but it doesn't satisfy my requirement. — umesh lohani, Jan 29 '20 at 09:37

Afsar edrisy · Answer 1 · 2020-01-30T10:05:52.740

Firstly there is no callback interface with MediaProjection which can give you buffer of screen captured frame, But it is quite possible with SurfaceTexture. Here is one implementation ScreenCapturerAndroid of screen capturing with SurfaceTexture.

An implementation of VideoCapturer to capture the screen content as a video stream. Capturing is done by MediaProjection on a SurfaceTexture. We interact with this SurfaceTexture using a SurfaceTextureHelper. The SurfaceTextureHelper is created by the native code and passed to this capturer in VideoCapturer.initialize(). On receiving a new frame, this capturer passes it as a texture to the native code via CapturerObserver.onFrameCaptured(). This takes place on the HandlerThread of the given SurfaceTextureHelper. When done with each frame, the native code returns the buffer to the SurfaceTextureHelper

But if you want to send stream of your app view then following screen capturer will help you.

public class ScreenCapturer {
    private boolean capturing = false;
    private int fps=15;
    private int width,height;
    private Context context;
    private int[] frame;
    private Bitmap bmp;
    private Canvas canvas;
    private View contentView;
    private Handler mHandler = new Handler();
    public ScreenCapturer(Context context, View view) {
        this.context = context;
        this.contentView = view;
    }

    public void startCapturing(){
        capturing = true;

        mHandler.postDelayed(newFrame, 1000 / fps);
    }
    public void stopCapturing(){
        capturing = false;
        mHandler.removeCallbacks(newFrame);
    }
    private Runnable newFrame = new Runnable() {
        @Override
        public void run() {
            if (capturing) {
                int width = contentView.getWidth();
                int height = contentView.getHeight();
                if (frame == null ||
                        ScreenCapturer. this.width != width ||
                        ScreenCapturer.this.height != height) {

                    ScreenCapturer.this.width = width;
                    ScreenCapturer.this.height = height;

                    if (bmp != null) {
                        bmp.recycle();
                        bmp = null;
                    }
                    bmp = Bitmap.createBitmap(width,
                            height, Bitmap.Config.ARGB_8888);

                    canvas = new Canvas(bmp);
                    frame = new int[width * height];
                }
                canvas.saveLayer(0, 0, width, height, null);
                canvas.translate(-contentView.getScrollX(), - contentView.getScrollY());
                contentView.draw(canvas);

                bmp.getPixels(frame, 0, width, 0, 0, width, height);


               //frame[] is a rgb pixel array compress it to YUV if want and send over RTMP

                canvas.restore();
                mHandler.postDelayed(newFrame, 1000 / fps);

            }
        }
    };


}

Usage

...
//Use this in your activity 

private View parentView;

parentView = findViewById(R.id.parentView);
capturer = new ScreenCapturer(this,parentView);

//To start capturing
 capturer.startCapturing();


//To Stop capturer
capturer.stopCapturing();

Using this you can send a view contents to RTMP stream, You may use parent view of your activity to capture all contents of an activity.

In Android Screen Recording - How can I get each frame?

1 Answers1