Android - Reading VirtualDisplay frames in real time

Question

I am capturing my Android device's screen using the MediaProjection API by creating a VirtualDisplay and displaying it on a SurfaceView that remains always visible. I would like to perform some image processing on the frames grabbed by the VirtualDisplay then draw them back to the SurfaceView, all in real time. Is there any way to do this with the VirtualDisplay and MediaProjection APIs?

RecordingSession.java

class RecordingSession implements SurfaceHolder.Callback {
  static final int VIRT_DISPLAY_FLAGS=
    DisplayManager.VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY |
      DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC;
  private RecordingConfig config;
  private final Context ctxt;
  private final ToneGenerator beeper;
  private MediaRecorder recorder;
  private MediaProjection projection;
  private VirtualDisplay vdisplay;
  private Window window;
  private Bitmap latestBitmap=null;

  RecordingSession(Context ctxt, RecordingConfig config,
                   MediaProjection projection, Window window) {
    this.ctxt=ctxt.getApplicationContext();
    this.window = window;
    this.config=config;
    this.projection=projection;
    this.beeper=new ToneGenerator(
      AudioManager.STREAM_NOTIFICATION, 100);
  }

  void start() {
    this.window = new Window(this.ctxt);
    this.window.open();
    this.window.getScreenShot().getHolder().addCallback(this);
    vdisplay=projection.createVirtualDisplay("andcorder",
            config.width*4, config.height*4, config.density,
            VIRT_DISPLAY_FLAGS, null, null, null);
    beeper.startTone(ToneGenerator.TONE_PROP_ACK);
  }

  void stop() {
    projection.stop();
    vdisplay.release();
    beeper.startTone(ToneGenerator.TONE_PROP_NACK);
  }

  @Override
  public void surfaceCreated(@NonNull SurfaceHolder surfaceHolder) {

  }

  @Override
  public void surfaceChanged(@NonNull SurfaceHolder surfaceHolder, int i, int i1, int i2) {
    vdisplay.setSurface(surfaceHolder.getSurface());
  }

  @Override
  public void surfaceDestroyed(@NonNull SurfaceHolder surfaceHolder) {

  }

  public void end() {
    this.window.close();
  }
}

score 0 · Answer 1 · answered Mar 13 '23 at 02:49

Yes, use the ImageReader class and use it's surface and submit that to the VirtualDisplay. The ImageReader will then start to call it's callback from the Images produced by the VirtualDisplay and push Images them to your listener.

You can then convert the Image to a Bitmap. The Bitmap can be drawn onto the render target surface by locking the canvas and drawing the Bitmap.

This will give you full control of the Bitmap. For example I'm drawing the Bitmap to the canvas and scaling it down so that all devices produce the same recorded movie width size. The slowest part here will be conversion of Image -> Bitmap which needs to grab the bytes from the GPU. But the drawing part will be done in GPU as long as you don't try do apply any drawing to the Bitmap.

Android - Reading VirtualDisplay frames in real time

1 Answers1